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Instructional  Systems  Development  (ISD) 
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This  study  presents  an  assessment  of  the  quality  of  graduates  from  instruc- 
tional systems  development  (ISD) -type  courses.  Performances  of  the  ISD-tvyie 
and  traditionally  instructed  (TI)  graduates  from  several  courses  were  com- 
pared. Differences  and  similarities  in  the  TI  and  ISD-type  curricula  were 
determined  by  comparing  course  content,  attrition  rates,  average  on  board 
and  per  capita  cost  to  train.  The  report  concludes  the  following:  ISD-tv^pe 

courses  produced  graduates  whose  quality  of  performance  was  equivalent  to  the 
graduates  of  TI  courses;  the  major  difference  between  ISD-tv-pe  and  TI 
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pacing often  results  in  a decrease  in  AOB;  ISD-t\pe  conversion  did  not  have 
a consistent  impact  on  attrition;  an  impact  on  per  capita  cost  to  train 
could  not  be  identified. 
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SUMMARY 


1.  This  report  was  prepared  in  response  to  a request  by  the 
Chief  of  Naval  Operations  (0P-964D)  to  assess  the  quality  of 
graduates  from  instructional  systems  development  (ISO)- type 
courses.  The  term  "ISD-type"  is  used  to  emphasize  that  at 
present  there  are  no  graduates  from  "ISD"  courses  (i.e.,  courses 
developed  by  the  Chief  of  Naval  Education  and  Training's  In- 
structional Program  Development  (IPD)  field  activities  using 
the  instructional  system  development  model  as  outlined  in 
NAVEDTRA  106).  The  Chief  of  Naval  Technical  Training  has  re- 
vised approximately  10  courses  using  a system-oriented  meth- 
odology similar  to  the  official  NAVEDTRA  model.  These  courses 
had  graduates  who  were  available  for  analysis.  The  study  was 
funded  under  Contract  N00014- 73-C- 008 1 . 

2.  The  Navy  is  planning  a large-scale  implementation  of  ISD 
courses,  but  at  present  there  is  no  feedback  system  that  eval- 
uates the  on-the-job  performance  of  ISD-type  graduates.  It 

is  believed  that  the  research  of  this  study  will  provide  an 
interim  assessment  of  the  on-the-job  performances  of  ISD-type 
graduates  as  compared  with  graduates  of  traditional  instruction 
(TI). 

3.  The  primary  objective  of  this  study  is  to  compare  the 
performances  of  the  ISD-type  and  TI  graduates  from  several  of 
the  same  courses.  The  secondary  objective  is  to  compare  the 
course  content,  attrition  rates,  average  on  board  (AOB),  and 
per  capita  cost  to  train  of  the  ISD-type  courses  with  the  TI 
courses  to  demonstrate  the  differences  and  similarities  in  the 
two  types  of  curricula. 

4.  The  ISD-type  courses  selected  for  analysis  required  a 
large  student  throughput  to  assure  the  availability  of  a large 
sample  for  study.  In  addition,  ISD-type  implementation  had 

to  have  taken  place  approximately  2 yr  ago  so  that  graduates 
would  have  had  opportunities  to  take  the  Third  Class  Advancement 
Examination.  The  four  courses  selected  were; 

• Avionics  Technician  (AV)  Class  A 

• Aviation  Electrician's  Mate  (AE)  Class  A 
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• Engineman  (EN)  Class  A 

• Machinists  Mates  (MM)  Class  A. 

5.  Two  cohorts  of  ISD-type  graduates  and  TI  graduates  were 
compiled  for  each  of  the  four  courses  using  the  training  records 
from  each  school.  When  TI  graduate  records  were  not  readily 
available,  the  ISD-type  graduates  were  compared  with  A-school 
graduates  from  the  same  course  or  all  members  of  those  rates 
within  the  respective  rating.  Data  were  collected  for  compar- 
ison from  the  following  three  sources: 

• Follow-on  training  courses 

• On-the-job  performance  marks  submitted  for 
promotion  to  Third  Class 

• Third  Class  Advancement  Examination  scores. 

FINDINGS  AND  CONCLUSIONS 

6.  The  analysis  of  the  comparison  of  ISD-type  and  TI  grad- 
uates yielded  the  following  results: 

a.  In  the  follow-on  Advanced  First  Term  Avionics 
(AFTA)  course,  the  test  results  of  ISD-type 
graduates  were  better  than  the  TI  graduates  but 
the  differences  in  final  course  grades  were  not 
statistically  significant. 

b.  In  the  follow-on  Naval  Air  Maintenance  Training 
Groups  (NAMTRAGRUs) , the  ISD-type  graduates  had 
higher  final  grades  than  the  TI  graduates  and 
all  other  students  taking  the  same  courses.  The 
differences  were  not  statistically  significant. 

c.  The  analysis  of  the  on-the-job  performance  marks 
submitted  for  promotion  to  Third  Class  indicated 
that  AE  and  EN  ISD-type  graduates  had  higher 
performance  marks  than  their  TI  counterparts. 

The  MM  graduates,  however,  had  lower  on-the- 

job  performance  marks  than  their  comparison  co- 
horts. The  differences  were  not  statistically 
significant . 

d.  The  analysis  of  the  scores  on  the  Third  Class 
Advancement  Examination  indicated  that  the  AE 
ISD-type  graduates  had  higher  scores  than  the 
TI  graduates.  The  EN  and  MM  ISD-type  graduates 
had  lower  scores  for  the  first  examination  and 
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higher  scores  on  the  second  and  third  exami- 
nations than  their  comparison  cohorts.  The 
MM  ISD-type  graduates  had  significantly  lower 
scores  on  the  first  of  three  examinations  analyzed. 

This  was  the  only  comparison  that  was  statisti- 
cally significant  and  indicated  a higher  TI  than 
ISD-type  performance. 

7.  In  summary,  of  35  ISD-type  and  TI  performance  comparisons 
analyzed  and  subjected  to  a 0.01  level  of  significance,  2 
comparisons  snowed  ISD-type  graduate  performance  superior,  1 
comparison  showed  TI  graduate  performance  superior,  and  32  showed 
no  performance  difference.  Therefore,  it  is  concluded  that  ISD- 
type  graduates  from  the  four  class  A courses  analyzed  had  per- 
formances equivalent  to  the  graduates  from  the  TI  version  of 
the  course. 


a.  The  TI  and  ISD-type  curricula  are  similar  for 
the  major  topics  instructed.  TI  curricula 
have  instructional  units  specifically  oriented 
to  theory.  ISD-type  curricula  do  not  consider 
theory  in  separate  units  but  rather  incorporate 
it  into  practical  instruction. 

b.  Implementation  of  ISD-type  curricula  with  self- 
pacing resulted  in  a decrease  in  the  AOB,  The 
reduction  in  AOB  for  the  four  courses  studied 
totaled  786. 

c.  The  academic  attrition  rates  for  all  courses 
ranged  from  0.9%  to  5.5%,  whereas  the  total 
attrition  had  a range  of  1.5%  to  11.2%. 

d.  The  influence  in  per  capita  cost  to  train  attri- 
butable to  a change  from  TI  to  ISD-type  courses 
could  not  be  identified. 
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I.  INTRODUCTION 


BACKGROUND 

1.1  This  study  was  funded  by  the  Chief  of  Naval  Operations 

as  a portion  of  the  work  performed  by  Presearch  Incorporated 
for  the  Systems  Analysis  Division,  Manpower/Per sonnel/Tr aining/ 
Reserve  Section  (OP-964D),  under  Contract  No.  NOOO 1 4 - 7 3- C- 008 1 . 
The  study  is  an  outgrowth  of  Presearch's  interim  report  of  50 
March  1976,  Instructional  System  Development  (ISP):  Review 

and  Appraisal , which  noted  that  a systematic  evaluation  of  grad 
uates  from  "ISD-type"  courses  was  needed. 

1.2  The  term  "ISD-type”  is  used  throughout  the  report  to  em- 
phasice  that  there  are  at  present  no  graduates  available  for 
analysis  from  "ISD"  courses  (courses  developed  by  the  Chief, 
Naval  Education  and  Training  (CNET],  Instructional  Program 
Development  (IPD),  and  implemented  at  field  activities  using 
the  ISD  model  as  outlined  in  NAVEDTRA  106).  Therefore,  to 
assess  the  quality  of  graduates  from  courses  that  were  revised 
using  a systems-or iented  methodology  and  seif-pacing  (and  there 
fore  representative  but  not  necessarily  duplicative  of  ISD 
course  graduates),  it  was  necessary  to  analyze  the  graduates 
from  ISD-type  courses.  The  approximately  10  courses  revised 

by  the  Chief  of  Naval  Technical  Training  (CNTT)  using  a systems 
approach  similar  to  the  official  NAVEDTRA  106  model  are  the 
best  courses  available  for  analysis  at  present. 
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PURPOSE 


1.3  There  is  no  functioning  structural  feedback  system  in 
operation  with  which  to  evaluate  the  postschool ing  performance 
of  training  course  graduates.  Therefore,  there  has  been 

no  systematic  assessment  of  the  on-the-job  performance  of  ISD- 
type  graduates  as  compared  to  traditionally  instructed  gradu- 
ates. It  was  believed  that  research  was  warranted  (even  if 
the  best  results  attainable  at  the  present  time  must  be  termed 
inferential  rather  than  conclusive)  because  of  the  large  scale 
of  ISO  implementation  planned  for  Navy  specialized  training. 

The  results  of  this  study's  research  will  be  useful  in  furnish- 
ing an  interim  checkpoint  assessment  of  the  quality  of  graduates 
from  self-paced  courses  developed  using  a systems-or iented 
methodology . 

OBJECTIVE  -•VND  SCOPE 

Objective 


1.4  The  primary  objective  of  this  study  is  to  compare  the 
postcourse  performance  of  graduates  from  several  ISD-type  courses 
with  the  performance  of  the  graduates  from  the  same  courses 
in  their  pre-ISD-type  configuration.  Pre-ISD-type  courses  have 
been  termed  traditional  instruction  (TI).  A secondary  objective 
is  to  compare  the  curricula  content  of  the  TI  and  ISD-type  ver- 
sions as  a basis  for  rationalizing  the  differences  (if  any) 
in  graduate  performance. 


1/ 


Evaluation  systems  are  currently  under  development  at  CNET 
and  CNETS. 
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1.5  As  explained  in  greater  detail  in  Section  III,  which  dis- 
cusses the  research  methodology  employed,  graduates  were  com- 
pared using  the  following  criteria: 

• Performance  in  follow-on  training  courses 

• Performance  on  the  Third  Class  Advancement  Exami- 
nation 

• Performance  marks  received  at  duty  stations 
following  training. 

Scope  and  Limitations 


1.6  The  ISD-type  courses  available  for  comparative  analysis 
are  limited  in  number  (approximately  10).  In  addition  to  that 
constraint,  the  courses  chosen  had  to  meet  several  criteria 
for  the  study  results  to  be  as  representative  as  possible. 
Specifically,  courses  from  the  aviation  and  nonaviation  com- 
munity should  be  included.  The  courses  chosen  had  to  have  ex- 
tensive study  throughput  to  provide  the  large  cohorts  required 
for  statistical  validity.  Lastly,  the  courses  chosen  had  to 
have  been  converted  from  TI  to  ISD-type  approximately  2 yr  ago 
to  facilitate  data  collection  and  to  permit  the  graduates  to 
have  had  an  opportunity  to  participate  in  rate  advancement  ex- 
aminat ions . 

1.7  It  was  determined  that  the  Class  A courses  for  Avionics 
Technicians  (AVs),  Aviation  Electricians  (AEs),  Machinists  Mates 
(MMs),  and  Enginemen  (E.Ns)  best  met  the  criteria,  and  they  were 
chosen  for  analysis. 
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l.S  Ef  f ect  I '>'icss  of  training  programs  can  be  measured  by 

graduate  quality,  an  area  of  major  concern  to  warfare  sponsors. 
Of  specific  concern  for  this  report  was  the  quality  of  the  ISD- 
type  graduates. 

1.9  An  evaluation  of  graduates  of  ISD-type  courses  required 
either  an  elaborate  survey  or  the  use  of  existing  data  compiled 
on  the  graduates  during  certain  points  in  their  careers.  Since 
the  use  of  existing  data  is  the  most  economical  approach,  ISD- 
type  courses  were  selected  that  had  the  most  existing  data, 
and  comparison  groups  were  chosen  that  had  the  best  available 
records.  Appropriate  statistical  methods  were  used  to  compare 
the  ISD-type  graduate  to  the  TI  graduate  or  to  established 
standards  for  his/her  respective  rate  and  rating. 

Career  Points  for  Evaluation 


1.10  The  career  points  for  evaluation  were  the  following: 

• B-schools  immediately  after  completing  the 
A- schools 

• Follow-on  courses  when  reporting  to  fleet 
or  shore  billets 

• An  average  of  the  graduates'  most  recent  perfor- 
mance evaluations  submitted  by  their  commands 
for  the  Third  Class  Advancement  Examination 
(on-the-job  performance  marks) 
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• Graduates'  performances  on  the  Third  Class 

1 / 

Advancement  Examination.  — 

1.11  The  sequence  for  choosing  the  graduate  samples  in  the 
comparison  of  ISD-type  graduates  was  as  follows: 

• ISD-type  cohort  to  TI  cohort 

• ISD-type  cohort  to  A-school  graduates  from 
the  same  course 

• ISD-type  cohort  to  all  members  of  those  rates 
within  the  respective  rating. 

Selection  of  ISP-Type  Courses 


1.12  The  CNTT  had  identified  several  courses  that  complied 
loosely  with  an  80^  or  greater  adherence  to  the  Interservice 
Procedures  for  Instructional  Systems  Development  (NAVEDTRA  106.\) 
model.  These  courses  were  investigated  to  determine  if  the 
students  graduating  from  them  could  be  evaluated  at  one  or  more 
of  the  career  points.  Four  ISD-type  courses  were  selected  for 
study.  Each  course  had  graduates  who  could  be  evaluated  at 
two  or  more  of  the  four  career  points.  The  four  courses  se- 
lected were  as  follows; 


• Avionics  Technician  (AV)  School,  Class  A 
(C- 100-2013) 

• .Aviation  Electr  id  ' s Mate  (AE  Course,  Class 
A (C-602-2012) 


2 / 

- .A-school  training  prepares  the  student  for  job  performance 
and  not  to  take  examinations.  It  is  recognited  that  some 
of  the  four  career  points  measure  the  graduates  with  written 
tests.  However,  if  the  ISD-type  graduate  can  be  evaluated 
at  all  four  of  the  career  points,  then  a statement  as  to  the 
quality  of  ISD-type  training  can  be  made. 
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• Engineman  (EN),  Class  A [A-652-0018) 

• Machinists  Mates  (MM),  Class  A (A-651-0015 ) . 

1.13  Table  1.1,  the  evaluation  matrix,  gives  a quick  reference 
of  all  evaluations  made  for  this  study.  Figure  1.1  illustrates 
the  education  pipeline  the  students  from  the  four  courses  follow 
from  recruit  training  to  the  Third  Class  Advancement  Examina- 
tion. 

Selection  of  Graduate  Comparison  Groups 


1.14  When  the  ISD-type  courses  were  selected,  visits  to  the 
schools  were  scheduled,  and,  with  the  help  of  the  respective 
training  offices,  data  were  collected  on  graduates.  Each  course 
and  its  graduates  presented  a different  problem.  The  develop- 
ment of  comparison  groups  or  cohorts  is  explained  in  the  fol- 
lowing paragraphs. 

1.15  AV  School . The  AV  school  was  using  the  TI  method  until 
January  1975,  with  students  graduating  as  late  as  March  1975. 

From  March  1975  until  September  1975,  the  AV  school  was  being 
changed  from  TI  to  the  ISD-type  course  that  was  self-pacing. 
Students  who  graduate  from  the  AV  course  and  have  a 6-yr  obli- 
gation are  sent  to  the  26-wk  Advanced  First  Term  Avionics  (AFTA) 
Course,  Class  B (C-  100/ 110-  2010 ) , given  at  Millington,  Tennessee, 
which  uses  TI  methods.  The  curriculum  for  this  course  has  not 
been  revised  during  the  past  2 yr . Two  cohorts  of  TI  and  ISD- 
type  graduates  of  the  AV  school  were  developed  to  measure  their 
performance  in  the  AFTA  school.  The  TI  cohort  consisted  of 

204  students  who  graduated  from  the  AV  course  from  January  to 
March  1975,  and  the  ISD-type  cohort  consisted  of  535  students 
who  graduated  during  April  and  May  1976.  The  ISD-type  cohort 
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TABLE  1.1 
EVALUATION  MATRIX 


Career 

Points 

ISD-Type 

Courses 

B-School  ! 

i 

Follow-on 

Courses 

Performance  ! 
Marks 

Third-Class 
Advancement 
Examinat i on 

Avionics 

Technician 

(AV) 

ISD-type 

versus 

TI 

ISD-type 

versus 

TI 

No 

Data 

No 

Data 

Aviation 
Electrician' s 
Mate 
(AE) 

No 

Courses  1 

ISD- type 
versus 
TI 

ISD- type 
versus 
TI 

ISD- type 
versus 
TI 

Engineman 

(EN) 

No 

Courses 

No 

Courses 

1 ISD-type 
versus 
All  E-3 
ENs 

ISD-type 
versus 
EN  A- School 
Graduates 

Machinists 

Mates 

(MM) 

No 

Courses 

! 

No 

Courses 

1 

1 

ISD- type 
• versus 
All  E-5 
MMs 

ISD- type 
versus 
MM  A- School 
Graduates 
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consisted  of  all  AV  graduates  who  completed  the  AFTA  course 
by  the  time  the  data  were  collected  in  November  1976.  This 
data  were  collected  so  that  the  cohorts  would  not  include  stu- 
dents from  the  transition  period. 

1.16  The  graduates  from  the  .AV  school  who  were  not  detailed 
to  the  AFT.A  course  (those  who  did  not  have  a 6-yr  obligation) 
were  sent  to  Naval  .Air  .Maintenance  Training  Groups  (NAMTRAGRUs) 
where  they  were  given  training  on  specific  aircraft  systems. 

Two  cohorts,  one  with  about  500  TI  graduates  and  one  with  ap- 
proximately 500  ISD-type  graduates,  were  developed  so  that 
performance  in  the  N.A.MTRAGRUS  could  be  measured.  .AFT.A  graduates 
were  not  included  in  these  cohorts. 

1.17  The  ISD-type  graduates  did  not  complete  the  .AV  schooling 
early  enough  to  be  eligible  to  take  the  Third  Class  .Advancement 
Examination;  therefore,  performance  could  not  be  compared  on 

the  examination  results  or  with  the  on-the-job  performance  marks. 

1.18  AE  Course.  The  AE  Class  .A  school  changed  curriculum  from 
TI  to  ISD-type  training  with  group  pacing  in  January  1975. 

All  students  who  started  the  course  after  7 January  1975  re- 
ceived ISD-type  training,  and  those  who  started  before  that 
date  received  TI  training. 

1.19  Two  cohorts  of  about  500  graduates  each  were  developed. 

The  AE  student  pipeline  did  not  have  an  immediate  follow-on 
B-school  course  comparable  to  the  AFT.A  course  for  .AVs,  but 
graduates  received  training  in  the  NA.MTRAGRU  before  they  re- 
ported to  fleet  or  shore  billets.  Each  of  the  cohorts  had 
graduated  early  in  1975  and  were  eligible  to  take  the  Third 
Class  .Advancement  Examination  in  August  1975  and  February  and 
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'T- 


August  1976.  Therefore,  the  AE  cohorts  could  be  compared  at 
three  career  points:  follow-on  training  (NAMTRAGRUs ) , on-the- 

job  performance,  and  Third  Class  Advancement  Examination. 

1.20  Propulsion  Engineering  (PE)  School.  The  PE  school  changed 
curricula  from  TI  to  ISD-type  in  January  1974.  The  three  courses 
given  at  the  PE  school  are  the  following: 

• Boiler  Technician  (BT),  Class 

• Engineman  (EN),  Class  A 

• Machinists  Mates  (MM),  Class  A, 

1.21  The  BT  course  was  not  included  in  this  study  at  the  re- 
quest of  the  C.NET.  Therefore,  only  EN  and  MM  graduates  were 

evaluated.  — The  PE  school  does  not  have  an  immediate  follow- 
on  B- school,  and  no  appropriate  follow-on  courses  were  available 
for  analysis  at  fleet  or  shore  stations.  When  the  PE  school 
changed  from  TI  to  ISD-type  training,  the  school  was  moved  into 
a new  building  that  was  designed  to  better  facilitate  the  sched- 
uling requirements  for  the  ISD-type  curricula.  To  conserve 
space,  the  PE  school  sent  all  records  of  the  Tl-trained  grad- 
uates to  the  record  files  at  St.  Louis,  Missouri;  hence,  only 
ISD-type  graduates'  records  were  readily  available  in  the  number 
required.  Therefore,  ISD-type  graduates  were  compared  to  all 
A-school  graduates  from  the  same  rate  and  rating  taking  the 
same  examination.  For  the  on-the-job  performance  marks  evalu- 
ation, the  ISD-type  graduate  was  compared  to  all  personnel  who 


— The  BT  Class  A course  in  the  opinion  of  CNET  (Dr.  Scanland) 
was  not  a suitable  candidate  for  this  research  because  of  the 
many  recent  perturbations  in  that  course  attributable  to  the 
installation  of  1,200-psi  ( 84 . 4 -kg/ sq- cm ) steam  equipment. 

4 / 

- The  EN  and  MM  cohorts  were  screened  to  avoid  including  any 
nuclear  students. 
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had  performance  marks  submitted  within  the  respective  rate  and 
rating.  Five  hundred  records  of  the  first  EN  and  MM  ISD-type 
graduates  were  used  to  make  the  comparisons. 

Statistical  Methodology 


1.22  It  was  assumed  that  each  cohort  was  normally  distributed 

2 

with  mean  p and  variance  a . If  the  two  cohorts  to  be  evaluated 
had  equal  variances,  then  the  analysis  of  variance  (.\NOVA)  test 
was  used  to  determine  the  difference  in  performance  of  the  cwo 
cohorts.  If  they  had  unequal  variances,  then  a two-tailed  t-test 
was  used.  The  F-ratio  test  was  employed  to  determine  if  the  two 
cohorts  had  equal  variance.  A level  of  significance,  a = 0.01, 
was  used  to  test  each  hypothesis.  For  more  details  on  the  sta- 
tistical analysis  methodology  employed,  see  Appendix  C. 

CONTENTS 

1.23  Section  II  briefly  contrasts  TI  and  ISD-type  methodologies 
and  then  provides  a comparative  analysis  of  curriculum  content 
for  the  courses  employed  in  this  study.  The  section  concludes 
with  a comparison  of  average  on  board,  attrition,  and  student 
cost  for  the  TI  and  ISD-type  versions  of  the  courses  researched. 
The  results  of  TI  and  ISD-type  student  performance,  using  co- 
horts drawn  from  graduates  for  four  Class  A courses  are  pre- 
sented in  Section  III.  Section  IV  summarizes  the  results  of 
the  study  and  presents  the  conclusions  that  represent  the  most 
significant  findings  of  the  research.  The  personnel  contacted 
during  the  course  of  the  research  are  identified  in  Appendix 

A,  the  significant  references  are  listed  in  Appendix  B,  and 
a description  of  the  statistical  procedures  employed  is  provided 
in  Appendix  C. 
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II.  CURRICULUM  REVIEW 


2.1  The  emphasis  in  management  and  systems  design  has  become 
"objective"  oriented.  This  approach  states  the  goal  and  then 
implements  the  appropriate  decision  making  and  designs  required 
to  reach  the  goal.  If  an  organization  or  action  does  not  have 
overall  "objective"  function,  then  it  is  eliminated.  Education 
has  also  implemented  the  "objective"  approach  to  curriculum 
development.  The  Navy,  to  modernize  its  educational  system 

in  the  same  way  as  the  civilian  educational  system,  is  moving 
from  traditional  instruction  (TI)  to  instructional  systems  de- 
velopment (ISO).  This  section  contrasts  the  general  instruc- 
tional methods  of  TI  and  ISO  by  evaluating  three  TI  and  ISD- 
type  curricula.  Average  on  board  (AOB),  attrition  rates,  and 
per  capita  costs  to  train  are  compared  for  TI  and  ISD-type 
courses. 

TI  .AND  ISD  COMP.ARISON 

2.2  TI  curricula  focus  on  a logical  progression  from  theory 

to  practical  applications.  ISD  focuses  on  skills  and  job  tasks, 
with  some  theory  included  to  support  the  skills.  The  following 
paragraphs  describe  in  greater  detail  the  major  characteristics 
of  the  TI  and  ISD  concepts. 

li 

2.3  TI  curricula  are  frequently  designed  to  progress  from 
basic  theory  to  the  practical  application  of  the  theory.  These 
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curricula  often  contain  topics  of  instruction  not  required  for 
job  task  performance.  The  curricula  and  tests  are  revised  pri- 
marily when  theories  are  added,  deleted,  or  changed. 

2.4  TI  is  almost  e.xclusively  group  paced.  Emphasis  is  on 
the  instructor  and  his/her  personal  involvement  in  the  instruc- 
tional process.  The  actual  method  of  instruction  relies  pre- 
dominantly on  lectures  with  diagram  instructional  aides.  Job 
tasks  are  taught  during  laboratory  periods,  which  are  scheduled 
approximately  1 day  a week. 

ISP 

2.5  ISO  is  a procedure  for  assuring  application  of  a systems 
approach  to  the  design,  implementation,  and  evaluation  of  the 
total  learning  process.  This  instructional  system  is  designed 
to  teach  personnel  the  knowledge  and  skills  necessary  to  per- 
form specific  jobs  with  minimal  time  and  cost.  ISD  curricula 
are  developed  in  the  five-step  process  that  follows: 

• Phase  One- -Analyze . Selection  of  tasks  for  in- 
struction by  evaluation  of  job  tasks;  estab- 
lishment of  performance  criterion. 

• Phase  Two--Design.  Development  and  sequencing 
of  learning  objectives  and  criterion  tests;  es- 
tablishment of  admission  standards. 

• Phase  Three- -Develop . Development  of  final  in- 
structional delivery  system;  establishment  of 
instructional  management  plans  for  resources 
(instructional  materials). 
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• Phase  Four- - Implement ■ Implementation  of  instruc- 
tional management  plan;  commencement  of  course 
and  instruction. 

• Phase  Five- -Control . Evaluation  of  instructional 
effectiveness  and  on-the-job  performance  in  fleet 
and  shore  billets;  revision  of  instructional 
system . 

2.6  The  analysis  and  design  phases  are  crucial  in  the  develop- 
ment of  ISD.  These  phases  identify  tasks  for  which  personnel 
are  expected  to  perform  in  fleet  and  shore  billets.  Learning 
is  accomplished  through  an  emphasis  on  the  active  effort  of 
the  student  rather  than  the  instructor.  Learning  objectives 
are  the  mode  used  to  express  precise  behavioral  objectives  prior 
to  selection  of  learning  activities.  A learning  objective  com- 
municates what  the  learner  will  be  doing  in  terms  of  terminal 
behavior  when  demonstrating  his  achievement.  To  write  a learn- 
ing objective,  the  following  information  must  be  stated  for 
each  objective; 

• Identification  of  the  overall  behavior 

• Definition  of  the  important  conditions  under 
which  the  behavior  is  to  occur 

• Definition  of  the  criterion  of  acceptable 
performance . 


Student  achievement  of  each  of  the  learning  objectives  is  mea- 
sured by  administering  tests  that  provide  direct  assessment 
of  whether  the  student's  performance  of  the  described  behavior 
under  the  predetermined  conditions  meets  the  criterion  specified 
in  the  learning  objectives.  A criterion  test  references  a stu- 
dent's achievement  to  the  learning  objective  criterion  rather 
than  rating  his  performance  against  others  being  tested. 
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2.7  Other  features  commonly  associated  with  ISO  are  learning 
centers,  student  self-pacing  (SP)  and  the  use  of  computer  tech- 
nology. A learning  center  contains  everything  a student  nor- 
mally needs  during  a course  of  instruction,  including  audio- 
visual devices  and  instructional  aides.  The  rationale  for  a 
learning  center  is  that  equipment  in  the  center  is  readily 
available  when  the  student  needs  it.  SP  refers  to  that  feature 
in  an  instructional  package  that  permits  the  individual  student 
to  attain  the  curriculum's  learning  objectives  at  his  own  learn- 
ing rate.  The  computer  instructional  technology  is  in  two  modes: 
computer-managed  instruction  (CMI)  and  computer-assisted  instruc- 
tion (CAI).  CMI  is  involved  with  student  progress  through  mod- 
ules and  testing;  with  CAI , the  computer  provides  instruction 
to  the  student . 


COMPARISON  OF  TI  .AN'D  ISD-TYPE  CURRICULA 


2.8  ISD-type  curricula  represent  an  intermediate  level  appli- 
cation of  ISO.  The  courses  studied  in  this  research  were  re- 
vised by  CNTECHTR.A  using  a systems-or iented  methodology  that 
resembles  the  ISO  approach  now  being  implemented  by  the  Chief, 
Naval  Education  and  Training  (CNET).  None  of  these  sample 
courses'  revisions  adherred  completely  to  the  official  ISD  model 
since  the  model  was  not  available  at  the  time  of  revision. 
Therefore  the  courses  researched  in  this  study  are  referred 

to  as  "ISD-type"  courses.  Specifically,  Phase  Five,  the  con- 
trol, evaluation,  and  revision  phase,  has  not  been  implemented. 
Course  evaluations  have  been  conducted,  but  no  formal  evaluation 
system  has  been  established. 

2.9  The  difference  in  instructional  approaches  is  evident 

by  comparing  several  school  curricula.  The  courses'  curricula 
are  evaluated  by  course  content  before  and  after  instituting 
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ISD-type  curricula.  Avionics  Technician  (AV)  School,  Aviation 
Electrician's  Mate  (AE)  School,  and  the  basic  Propulsion  Engi- 
neering (PE)  modules  for  the  Engineman  (EN)  and  Machinists  Mates 
(MM)  courses  are  used  to  compare  TI  and  ISD-type  curricula. 

EN , MM,  and  Boiler  Technician  (BT)  schools  commence  with  i3 
modules  of  basic  PE  instruction.  The  EN , MM,  and  BT  schools' 

TI  course  outlines  were  not  available;  therefore,  the  basic 
PE  module  curriculum  outlines  are  utilized  for  the  comparison. 

2.10  Identification  of  the  format  used  in  outlining  course 
content  and  mission  is  necessary  to  compare  TI  and  ISD-type 
curricula.  The  TI  curricula  of  the  AV  and  .AE  schools  and  the 
basic  PE  modules  are  designed  in  syllabus  form,  introduced  by 
a statement  of  course  mission.  The  topics  for  instruction  are 
listed  in  order  of  presentation.  The  ISD-type  curriculum  out- 
lines contain  a series  of  learning  objectives. 

2.11  There  are  constraints  in  comparing  TI  syllabus  topics 

to  ISD-type  learning  objective  "behavior  acts."  The  term  "over- 
all behavior"  is  sufficient  to  identify  the  final  main  task 
that  is  to  be  learned  when  there  are  actually  several  prereq- 
uisite skills  that  must  be  learned  and  performed.  The  overall 
behavior  acts  are  listed  in  the  learning  objectives,  whereas 
often  the  theory  objectives  are  not.  It  was  difficult  to  deter- 
mine the  exact  extent  to  which  this  has  occurred.  Learning 
objectives  are  used  in  the  following  discussion  to  infer  a syl- 
labus when  necessary. 

AV,  Class  A 


2.12  The  AV  A course  provides  the  skills  and  knowledge  to  per- 
form job  entry  avionics  maintenance  tasks  common  to  the  .Aviation 
Electronics  Technician  (AT),  Aviation  Fire  Control  Technician 
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CAQ)  and  Aviation  Antisubmarine  Warfare  Technician  (AX)  ratings. 
The  TI  course  had  an  average  completion  time  of  311.7  contact- 
hours  (15.4  wk) . The  ISD-type  course  has  an  average  completion 
time  of  485.5  contact -hours  (14.6  wk) . Of  these  total  ISD- 

type  contact-hours,  385  are  spent  in  the  learning  center  and 
62 0 are  spent  in  the  laboratory. 

2.13  The  TI  curriculum  data  source  is  "Curriculum  Outline  for 
.Avionics  Technician  Course,  Class  A,"  dated  ~ July  1970.  This 
document  is  a syllabus  preceded  by  a "course  mission,"  or  gen- 
eral learning  objective.  The  ISD-type  course  data  source  is 
"Curriculum  Outline  for  Avionics  Technician  Course,  Class  Al," 
dated  February  1976.  The  ISD-type  curriculum  outline  contains 
a syllabus  in  addition  to  very  specific  learning  objectives. 
Syllabi  were  used  to  compare  AV  course  TI  and  ISD-type  curric- 
ula. Although  individual  topics  within  these  units  may  differ 
by  content  and  sequencing,  the  analysis  has  been  limited  to 
the  main  unit  titles,  as  compared  in  Table  2.1. 

AE , Class  A 

2.14  The  mission  of  the  AE  course  is  to  provide  students  with 

the  knowledge  and  skills  required  to  perform  maintenance  in 

representative  naval  aircraft  e lec t r ica 1/e lectronic/ ins t rument 

systems.  The  TI  course  was  designed  to  consist  of  478.3  contact- 

•>  / 

hours  (14.4  wk) . - The  ISD-type  course  is  366.2  contact -hours 
(11  wk)  . 


One  contact-period  equals  50  min.  Eight  periods  per  day  for 
5 days  per  week  equals  33.53  contact -hours  per  week. 

1 / 

- The  curriculum  included  about  112.1  contact -hours  (3.4  wk) 
of  BE/E  and  566.2  contact -hours  (.11  wk)  of  .AE  curriculum. 


A 
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2.15  The  TI  curriculum  data  source  used  was  "Curriculum  Outline 
for  Aviation  Electrician's  Mate  School,  Class  A,"  dated  9 April 
1974.  The  ISD-type  curriculum  data  source  was  "Course  Outline 
for  Aviation  Electrician's  Mate  Course,  Class  Al,"  dated  20 
February  1976.  The  TI  outline  is  a syllabus;  the  ISD-type  out- 
line is  a series  of  course  learning  objectives.  The  major  units 
instructed  for  TI  and  ISD-type  curricula  are  compared  in  Table 
2.2. 

2.16  The  TI  curriculum  differs  from  ISD-type  in  two  topical 

areas:  introduction  and  troubleshooting.  The  TI  introductory 

topics,  such  as  safety,  hand  tools,  and  designs  and  schematics, 
have  been  eliminated  from  the  ISD-type  curriculum  as  a separate 
unit.  These  introductory  topics  are  presumed  to  be  incorporated 
into  the  ISD-type  curriculum,  without  necessitating  separate 
learning  objectives.  The  TI  syllabus  contains  a sentence  stat- 
ing that  troubleshooting  is  emphasized  throughout  the  course. 

The  ISD-type  learning  objectives  specify  troubleshooting  as 

a main  unit  in  addition  to  being  a knowledge  requirement  for 
each  topic. 

Basic  PE  Module 


2.17  The  PE  school  provides  the  skills  and  knowledge  required 
to  continue  training  in  the  ratings  of  EN , MM,  and  BT.  Each 
of  these  courses  is  preceded  by  15  modules  of  basic  PE  instruc- 
tion. These  modules  are  used  to  compare  TI  and  ISD-type  cur- 
ricula. The  TI  curriculum  consisted  of  a minimum  of  30  hr  per 
week  for  9 wk  of  instruction.  The  curriculum  outline  states 
the  students  should  receive  a combination  of  classroom  and  prac- 
tical work  daily.  The  self-paced  13-module  ISD-type  curriculum 
is  to  be  completed  in  approximately  126  contact -hour s (3.7  wk).  — ^ 

— ^ Each  class  day  has  about  6.75  contact-hours. 
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TABLE  2.2 

COMPARISON  OF  TI  .AND  ISD-TYPE  AE  CURRICULA 


TI  Unit 
Introduction 

Basic  electronics  applicable 
to  aircraft 

Electrical  machinery  devices 
and  circuits 

Aircraft  instruments  and 
flight  stabilization  system 

Aircraft  system  and  line 
maintenance 

None 


ISP-Type  Unit 

None 

Basic  electrical/electronic/ 
mechanical  repair 

Signal  tracing  of  basic  elec- 
tronic circuits 

Aircraft  electrical,  electronic, 
and  instrument  systems 

Aircraft  maintenance 
Troubleshoot ing 
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2.18  The  TI  curriculum  data  source  is  N'AVPERS  95799.  The 

EN , MM,  BT  courses'  TI  curricula  were  not  available  for  this 
study.  In  lieu  of  these  curriculum  outlines,  basic  PE  course 
data  were  used.  The  test  book.  Propulsion  Engineering  — ^ was 
used  to  infer  ISD-type  curriculum  content.  By  using  this  test 
book,  the  main  units  of  study  could  be  determined.  However, 
it  did  not  permit  the  determination  of  the  amount  of  introduc- 
tory material  provided  to  the  ISD-type  student.  The  exclusion 
of  certain  topics  in  the  test  book  infers  their  deletion  from 
the  ISD-type  curriculum.  However,  some  of  the  theory  topics 
may  have  been  incorporated  into  the  ISD-type  units  without  re- 
quiring specific  learning  objectives.  Topics  included  in  the 
basic  PE  modules  are  listed  in  Table  2.5. 

AVER.AGE  ON'  BOARD  (AOB) 

2.19  Implementation  of  ISD-type  curricula  has  had  an  impact 
on  course  length  and  AOB  as  a result  of  SP  and  curriculum  re- 
vision. The  revisions  of  curricula  have  reduced  the  course 
content  solely  to  that  material  that  is  necessary  to  perform 
a job  or  skill. 

2.20  Table  2.4  provides  throughput,  course  length,  and  AOB 
data  for  the  four  sample  courses,  i.e.,  AV,  .AE,  EN,  and  MM  and 
six  curricula  (pre-  and  post- ISD- type  implementation).  The 

MM  rating  has  two  curricula:  the  600-psi  ( 4 2 . 2 - kg/sq- cm)  and 


4 / 

- Bureau  of  Naval  Personnel,  Curriculum  for  Basic  Propulsion 
Engineering,  Class  A,  NAVPERb  95 /9i),  September  1909. 

— ^ U.S.  Government  Printing  Office,  Propulsion  Engineering, 
"Individualized  Learning  Svstem,"  third  Edition  I Progress 
Checks,  Modules  One  Through  Thirteen)  , 650-490  ' 1 402  , August 
1975  . 
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TABLE  2.3 

COMP.ARISON  OF  TI  AND  ISD-TYPE  BASIC  PE  CURRICULA 


TI  Unit 

Introduction,  indoctrination 

Introduction  to  marine  power 
plants 

Introduction  to  boilers 

Combustion  principles 

Internal  combustion  engines 
(including  gas  turbines) 

Mathematics  review 

Mechanical  theory 

Metallurgy 

Fluid  theory 

Electrical  theory 

Heat  theory 

Logs,  blueprints,  and  tele- 
phones 

Measuring  and  control  devices 

Power  transfer  equipment 

Hand  tools,  instruments,  and 
mater ials 

Valves,  piping,  and  fittings 

Pumps 

None 

None 

None 


ISP-Type  Unit 

None 

None 

None 

None 

None* 

None 

None 

None 

None 

None 

None 

None 

Precision  measuring  instruments 
and  technical  manuals 

Turbines  (steam),  couplings, 
and  gears 

Metal  fasteners  and  hand  tools 

Valves;  packing,  gaskets,  and 
insulation 

Pumps 

Bearings  and  lubrication 

Strainers  and  oil  purifiers 

Low-pressure  air  systems  and 
low-pressure  air  compressor 


* There  is  no  gas  turbine  instruction  in  the  ISD-type  curriculum. 
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the  1,200-psi  (.84 . 4-kg/sq-cm)  engine  curricula.  The  AV  course 
trains  four  ratings:  AT,  AQ,  AX,  and  TD.  The  duration  of  AT, 

AQ,  and  AX  courses  are  identical;  the  TD  course  is  shorter  than 
the  .AT,  .AQ,  and  .AX  courses.  Implementation  of  ISD-type  cur- 
ricula resulted  in  a decrease  in  course  length  for  five  of  the 
si.x  curricula.  Due  to  a lack  of  funds,  the  .AE  course  was  the 
only  course  of  the  six  that  did  not  develop  an  SP  curriculum, 
and  the  course  length  remained  at  11  wk. 

2.21  A reduction  in  course  length  resulted  in  a reduction  of 
the  projected  FY  1977  AOB.  .A  significant  financial  savings 
in  training  costs  can  occur  as  a result  of  the  reduction  in 
.AOB  caused  by  ISD-type  SP  implementation.  The  reduction  in 
length  of  the  four  curricula  had  the  effect  of  reducing  786 
student  billets.  — ^ 

ATTRITION 

2.22  .Academic  attrition  rates  are  evaluated  for  TI  and  ISD- 
type  curricula  to  (a)  quantify  the  degree  of  change  and  (b) 
determine  if  the  academic  attrition  is  in  reasonable  accordance 
with  Navy  expectations.  Table  2.5  presents  the  attrition  rate 
data  for  the  TI  and  ISD-type  AV,  AE,  EN,  MM,  and  basic  PE  cur- 
ricula. Both  academic  and  nonacademic  attritions  are  displayed 
by  attritions  per  thousand. 

2.23  The  data  reveal  an  increase  in  the  academic  attrition 
rate  for  ISD-type  curricula  for  the  AV  and  basic  PE  courses, 
a decrease  for  the  EN  and  MM  courses,  and  no  change  for  the 
AE  course.  Therefore,  an  ISD-type  conversion  did  not  have  a 


— The  AOB  reduction  of  80.2  for  the  .AV  school  was  of  a tempo- 
rary nature  because  of  a shorter  than  planned  course  caused 
by  absence  of  equipment. 
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TABLE  2.5 

ATTRITIONS  PER  THOUSAND 


Course 

TI  Course 

ISD- 

Type  Course 

•Academic 

1 Non- 
1 academic 

Total 

•Academic 

Non- 

academic 

Total 

AV 

52 

79 

55 

56 

AE 

20 

70 

20 

30 

Basic  PE 

5 

59 

42 

9 

63 

72 

EN* 

1 

1 14 

15 

1 

14 

NLM* 

2 

7 7 

24 

0 

64 

64 

* The  attritees  listed  for  the  basic  PE  module  are  from  the 
preliminary  5-wk  PE  curriculum  for  BT , EN , and  \LM  students. 
The  attritees  from  the  EN  and  MM  courses  are  drops  that 
occurred  subsequent  to  the  basic  PE  module. 
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consistent  impact  on  attrition.  The  BT,  EN,  and  MM  courses 
are  preceded  by  a 3-wk  module  of  basic  PE.  The  attritees  listed 
for  the  basic  PE  module  are  from  the  3-wk  module.  The  attritees 
from  the  EN  and  MM  courses  are  drops  that  occurred  subsequent 
to  the  basic  PE  module.  It  appears  that  in  the  transition  to 
ISD-type  curricula,  the  academic  attrition  for  the  EN  and  MM 
courses  was  shifted  to  the  basic  PE  module.  The  academic  attri- 
tion rate  at  EN , MM,  and  basic  PE  courses  is  less  than  1%,  which 
seems  to  be  a reasonable  rate.  The  1%  and  51  academic  attrition 
rates  for  the  AE  and  AV  courses,  respectively,  are  also  reason- 
able rates  for  academic  attrition. 

PER  CAPITA  COST  TO  TRAIN 

2.24  The  major  emphasis  of  the  research  in  this  study  is  in 
analyzing  the  quality  of  students  graduating  from  the  four  sam- 
ple Class  A courses  in  the  ISD-type  mode  as  compared  to  the 

TI  mode.  This  section,  however,  has  addressed  other  comparative 
aspects:  curriculum  content,  AOB,  and  attrition.  The  section 

concludes  with  an  analysis  of  the  cost  trends  for  the  four  sam- 
ple courses  covering  the  FY  1973  to  1976  period. 

2.25  At  the  present  time,  two  course  cost  reports  are  available 
for  analysis.  Both  are  compiled  by  CNTECHTR.4.  The  first,  com- 
piled monthly,  is  known  as  the  Monthly  School  Cost  Report. 

This  report  does  not  include  investment  costs  or  all  military 
personnel  allowance/benefit  costs.  Therefore,  while  it  is  time- 
ly, it  understates  actual  costs  significantly.  The  second  re- 
port, Per  Capita  Cost  to  Train,  has  investment  costs  and  all 
military  personnel  allowance/benefit  costs  included.  While 
it  is  the  most  complete  data  currently  available  for  analysis, 
it  does  not  include  all  equipment  costs  (training  equipment 
is  categorized  into  many  classifications  and  budgets,  which 
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encumbers  per  capita  training  cost  accounting),  nor  does  it 
include  most  computer  costs.  Per  Capita  Cost  to  Train  also 
has  a limitation  in  its  use  of  year-to-year  cost  trend  analysis 
in  that  it  is  computed  only  on  an  annual  basis.  As  CNTECHTRA 
completely  amortizes  investment  costs  in  the  year  in  which  they 
occur,  year-to-year  per  capita  cost  perturbations  occur  that 
are  attributable  to  the  uneven  flow  of  investment.  — 


Cost  Data  for  Sample  Courses 

2.26  The  per  capita  cost  to  train  for  the  four  sample  courses 
used  in  this  analysis  is  shown  in  Table  2.6.  A major  component 
in  the  per  capita  cost  is  student  MPN.  Three  of  the  four  courses 
had  a reduction  in  AOB  after  the  course  was  changed  to  an  ISD- 
type,  and  one  course  had  no  change  in  .AOB  (see  Table  2.4). 
Therefore,  the  student  MPN  cost  component  should  provide  a down- 
ward bias  in  the  per  capita  cost  to  train.  On  the  other  hand, 
the  data  in  Table  2.6  are  in  current  dollars  and  therefore  con- 
tained an  upward  bias  attributable  to  inflation.  The  investment 
flow  contains  high-value  equipment  costs  and  construction  costs; 
these  costs  were  components  particularly  noticeable  in  the  FY 
1976  EN  and  MM  A course  costs  because  of  improvements  made  in 

PE  facilities  that  year. 

2.27  It  can  be  seen  in  Table  2.6  that  the  trends  in  per  capita 
cost  to  train  for  the  four  sample  courses  varied  widely.  The 


7 / 

— The  limitations  of  the  present  costing  proceaures  are  recog- 
nized, and  CNET  has  promulated  CNET  Inst.  7310. 2A  (Subject: 
CNET  Training  and  Education  Costing  and  Report  System 
(TAECARS))  to  improve  costing  procedures.  It  is  anticipated 
that  the  improvement  effort  may  take  several  years  to  become 
fully  implemented.  In  the  interim  it  will  be  difficult  to 
isolate,  for  purposes  of  analysis,  any  one  cost  influence 
such  as  ISD  unless  some  modification  is  made  to  present  cost 
procedures . 
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TABLE  2.6 

PER  CAPITA  COST  TO  TRAIN 


Course 

Cost,  S 

FY  1975 

FY  1974 

FY  1975 

FY  1976 

.AE  A-1  (C-602-  2012  ) 

5,966.75 

5,205.64 

5,451.80 

AV  A-1  (C-100-2015) 

4,195.44 

5,975. 74 

4,575.58 

4,405.22* 

EN  A-1  (A-652-0018) 

1,919.06 

1,085. 55* 

852.55** 

MM  A-1  (A-651-0015) 

1,656 .18 

1,508.55* 

852 . 55** 

Source:  CNTECHTRA,  "Per  Capita  Cost  to  Train  Reports;  FY  75- 

FY  76,”  CNET  N-6215. 

* The  commencement  time  when  all  students  under  instruction  were 
matriculating  in  an  ISD-type  course. 

**  In  FY  1975  , the  EN  A and  NLM  A courses  were  costed  aggregately 
because  of  the  basic  PE  module  common  to  both.  Therefore,  the 
costs  were  shown  to  be  the  same. 
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two  aviation  courses  increased  nominally  in  FY  1976  after  con- 
version to  ISD-type,  but  the  increase  was  in  the  range  that 
could  be  attributed  to  inflationary  bias.  The  two  PE  courses 
increased  in  FY  1976  compared  to  FY  1974  (when  the  courses  were 
converted  to  ISD-type)  but  the  increases  differed  greatly  in 
magnitude- -EN  A course  cost  more  than  doubling  (presumably  be- 
cause of  investment  construction  costs),  while  MM  A course  cost 
remaining  almost  unchanged. 

2.28  The  present  cost  data  system  does  not  permit  any  high 
confidence  conclusion  to  be  made  regarding  the  effect  on  per 
capita  cost  attributable  to  a change  in  a given  course  from 
TI  to  ISD-type. 

CON'CLUSION 

2.29  The  TI  and  ISD-type  curricula  are  similar  for  the  majority 
of  the  topics  instructed.  While  it  is  not  possible  to  determine 
if  the  amount  of  theory  taught  is  different,  it  appears  that 

the  emphasis  on  theory  has  changed.  TI  curricula  commenced 
with  a great  deal  of  theory.  ISD-type  curricula  do  not  teach 
theory  as  separate  units  but  rather  incorporate  theory  with 
practical  skills. 

2.50  Implementation  of  ISD-type  curricula  has  resulted  in  a 
decrease  in  training  AOB.  The  reduction  in  training  for  the 
four  courses  studied  total  786  man-years. 

2.31  Academic  attrition  rates  varied  in  ISD-type  curriculum. 

An  increase  in  the  academic  attrition  rate  for  the  ISD-type 
curriculum  occurred  in  the  AV  and  basic  PE  courses,  a decrease 
occurred  for  the  EN  and  MM  courses,  and  no  change  occurred  in 
the  AE  course.  The  academic  attrition  rates  for  all  courses 
were  in  the  0.9a  to  5.5a  range. 


29 


ARCH 


2.32  The  influence  in  per  capita  cost  to  train  attributable 
to  a change  from  TI  to  ISD-type  could  not  be  isolated.  There- 
fore, no  conclusion  can  be  made  based  on  cost  data  in  the  pres- 
ent reports. 
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III.  GRADUATE  QUALITY  ANALYSIS 

3.1  The  analysis  of  the  instructional  systems  development 
(ISD)  type  graduates  was  conducted  at  four  points  and  compari- 
sons were  made  as  follows: 

a.  Immediate  follow-on  B-school:  ISD-type 

versus  traditional  instruction  (TI)  Avionics 
Technician  (AV),  Class  A school  graduates 

b.  Follow-on  functional  training  at  Naval  Air 
Maintenance  Training  Group  (NAMTRAGRU)  schools 
for : 

1.  ISD-type  versus  TI  graduates  for  AV 

2.  ISD-type  versus  all  other  students  in 
the  same  NAMTRAGRU  courses 

c.  Fleet  and  shore  duty  on-the-job  performance 
marks,  when  recommended  for  Third  Class: 

1.  ISD-type  versus  TI  for  AEs 

2.  ISD-types  versus  all  candidates  from  the 
respective  rate  and  rating,  for  Engineman 
(EN)  and  Machinists  Mates  (NLM) 

d.  Third  Class  Advancement  Examination: 

1.  ISD-type  versus  TI  graduates  for  AEs 

2.  ISn-tvpe  versus  all  A-school  graduates 
from  the  same  course,  for  ENs  and  MMs . 
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PERFORMANCE  IN  B- SCHOOL 

3.2  The  graduates  of  the  Avionics  Technician  (AV)  School  who 
have  6-yr  obligations  are  sent  to  the  Advanced  First  Term  Avi- 
onics (AFTA)  Course,  Class  B (C-  1 00/ 1 1 1 - 2010 ) . AFTA  students 
are  graded  in  four  areas: 

a.  Practical  work 

1.  Nine  grades  assigned 

2.  Comprises  of  final  grade 

3.  Grade  is  based  on  quality  of  work,  safety, 
completeness  and  time  required  to  complete 
work 

b.  Performance  tests 

1.  Nine  grades  assigned 

2.  Comprises  19%  of  final  grade 

3.  Graded  in  the  same  manner  as  the 
practical  work 

c.  Written  tests 

1.  Thirteen  grades  assigned 

2.  Comprises  63%  of  final  grade 

d.  Comprehensive  examination 

1.  Comprises  10%  of  final  grade 

2.  Given  at  the  end  of  the  course. 

3.3  The  quality  of  graduate  performance  is  related  to  the 
quality  of  student  input  to  that  curriculum.  If  student  input 
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quality  changes,  then  it  would  be  expected  that  graduate  per- 
f ormance  would  also  change.  To  determine  if  student  input 
quality  changed,  the  Basic  Test  Battery  (BTB)  scores  required 
for  selection  to  the  AV  school  were  compared  for  the  TI  and 
ISD-type  cohorts.  The  two  BTB  aptitude  tests  used  for  AV  Class 
.A  entrance  screening  are  the  Arithmetic  Reasoning  (ARI)  and 
Electronics  Technician  Selection  Test  (ETST).  The  required 
score  for  selection  is  171  or  higher  (ARI  and  2 ETST). 

Hypotheses 

3.4  The  hypotheses  tested  were  as  follows; 

• There  was  no  difference  between  the  mean  scores 
of  the  ISD-type  and  Tl-trained  students  in  the 
AFT.4  course.  This  hypothesis  was  tested  on  each 
of  the  five  test  scores  (practical  work,  per- 
formance test,  written  test,  comprehensive  exami- 
nation, and  final  course  grade). 

• There  was  no  difference  in  the  quality  of  the 
ISD-type  and  TI  students  selected  for  the  .AFTA 
course. 

Data  and  Data  Collection 


3.5  The  following  data  were  collected  cn  each  student: 

• Practical  work  raw  scores 

• Performance  test  raw  scores 

• Written  test  raw  scores 

• Comprehensive  examination  raw  score 
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• Final  course  grade 

• BTB  scores,  including  ARI  and  ETST. 

Statistical  Procedures 

5.6  The  variances  of  the  ISD-type  and  TI  cohorts  were  compared 
using  the  F-ratio  test.  The  variances  were  found  to  be  unequal. 
The  two-tailed  t-test  was  used  to  test  the  hypotheses.  A sig- 
nificance level  of  a = 0.01  was  selected  as  the  critical  de- 
cision value  (see  Appendix  C for  details). 

3.7  Table  3.1  illustrates  the  results  of  the  analysis: 

a.  Column  1 identifies  the  test  categories 

b.  Column  2 shows  the  percentage  of  the  final  course 
grade 

c.  Column  3 lists  the  means,  standard  deviations, 
and  sample  sites  of  the  ISD-type  cohort 

d.  Column  4 lists  the  means,  standard  deviations, 
and  sample  sites  of  the  TI  cohorts 

e.  Column  5 shows  the  t-statistic 

f.  Columns  6 and  7 show  that  significance  at  the 

a = 0.01  and  a = 0.05  levels  is  either  confirmed 
or  negated, 


— The  t-values  at  the  top  of  Columns  6 and  7 designate  the 
point  at  which  the  statistic  would  become  significant  at 
that  level.  If  the  absolute  value  of  the  t-statistic  is 
less  than  the  value  at  the  top  of  the  a columns,  then  there 
is  no  significant  difference  at  the  mean  grade  of  the  two 
cohorts.  If  it  is  equal  to  or  greater  than  the  t-value, 
there  is  a significant  difference. 
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Total  of  204  stiuionts. 
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Results 


5.8  The  results  of  this  portion  of  the  study  were: 

a.  The  ISD-type  students  had  higher  mean  scores  than 
the  Tl-trained  students  on  all  four  test  areas. 

b.  In  the  examination  areas  of  performance  tests 

and  practical  work,  the  mean  scores  of  the  ISD-type 

students  were  significantly  better  than  those  of 

? / 

the  Tl-trained  students.  — 

c.  The  ISD-type  students  scored  higher  on  the  15 
written  tests  and  the  comprehensive  examination 
(but  not  statistically  higher). 

d.  The  ISD-type  students  had  higher  mean  scores  on 
the  final  course  grade  (but  not  significantly 
higher) . 

e.  The  ISD-type  students  did  significantly  better 
on  two  of  the  four  test  areas  (practical  work 
and  performance  tests)  that ' compr i se  the  final 
course  grade;  however,  they  did  not  do  signifi- 
cantly better  on  the  final  course  grade.  — ^ 

f.  The  BTB  scores  show  that  student  input  to  the 

4 / 

ISD-type  course  had  higher  scores.  — 

— This  may  be  because  of  the  difference  in  curriculum  content 
emphasis  between  TI  and  ISD-type  curricula.  The  ISD-type 
curriculum  was  developed  by  defining  those  job  skills  the 
graduates  are  required  to  know  to  perform  the  day-to-day 
assignments  in  fleet  or  shore  billets.  As  a result,  the  ISD- 
type  curriculum  puts  greater  stress  on  the  hands-on  approach 
to  learning  job  skills. 

— ^ .N'ote  that  the  two  test  areas  only  accounted  for  27»  of  the 
final  course  grade. 

— ^ The  difference  was  not  significant  at  the  a = 0.01  level. 
There  was  a significant  differe-'ce  at  the  a = 0.05  level. 
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Conclusions 


3.9  The  conclusions  for  this  portion  of  the  study  were  based 
on  significance  level  of  a = 0.01.  They  are  as  follows; 

• There  was  no  significant  difference  between 
the  performances  of  the  ISD-type  students  and 
the  TI  students  in  the  AFTA  course 

• There  was  no  significant  difference  between  the 
ISD-type  and  TI  students'  BTB  cohort  scores. 

PERFORMANCE  IN  FUNCTIONAL  FOLLOW-ON  SCHOOLS 

5.10  When  Avionics  Technicians  (AVs)  and  Aviation  Electrician's 
Mates  CAEs)  are  assigned  to  aircraft  squadrons  in  the  fleet 
after  completing  A-school  training  at  Millington,  Tennesse, 
they  are  given  additional  training  at  one  of  the  Naval  Air  Main- 
tenance Training  Groups  (N.A.MTRAGRUS ) . This  portion  of  the  study 
presents  the  analysis  of  the  ISD-type  and  TI  graduates'  perfor- 
mances at  functional  follow-on  schools. 

Hypotheses  ' 


3.11 


The  hypotheses  tested  were  as  follows; 


5/ 


— Two  hypotheses  were  used  to  facilitate  testing;  (a)  ISD- 
types  against  TI  graduates  and  (b)  ISD-types  against  all 
other  non-TI  students.  The  reason  for  dual  testing  is 
that  N.A.MTRAGRUS  offer  many  courses.  The  original  Class  A 
school  cohorts  became  very  fragmented  at  this  point  in  the 
training  pipeline  (i.e.,  under  10,  as  can  be  seen  in  Table 
5.2).  Therefore,  the  ISD-types  were  compared  with  both  the 
TI  group  and  with  all  others. 
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0 There  was  no  difference  between  the  mean  final 
grades  of  the  ISD-type  and  TI  students  in  the 
NAMTRAGRU  courses. 

0 There  was  no  difference  between  the  mean  final 
grade  scores  of  the  ISD-type  graduates  and  other 
students  in  the  same  .N'.AMTRAGRU  courses. 

Data  and  Data  Collection 


5.12  To  keep  the  cost  of  data  collection  to  a minimum,  two 
east  coast  NA.MTR.AGRUs  were  selected.  They  were  Oceana,  Vir- 
ginia, where  the  required  data  could  be  collected  during  a 1- 
day,  on-site  visit  and  Cecil  Field,  Florida,  where  the  informa- 
tion was  in  a computerized  data  base  from  which  it  could  be 
extracted . 

3.15  AV  and  AE  Cohorts.  The  original  cohorts  collected  at 
Millington  included  Marine,  foreign,  and  reserve  students  along 
with  Navy  students.  When  these  non-Navy  students  were  deleted, 
the  cohorts  were  approximately  reduced  as  follows: 

0 AV  (ISD-type);  500  to  458 
0 AV  (TI);  700  to  634 

0 AE  (ISD-type):  500  to  546 

0 AE  (TI):  500  to  392. 

3.14  Oceana  Data.  The  final  course  grades  for  each  member 
of  the  four  cohorts  were  collected  from  the  NA.MTRAGRU  course 
graduation  rosters.  It  was  discerned  that  only  four  courses 
provided  sufficiently  large  sample  sizes  for  comparison.  The 
four  courses  were: 
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a.  ARN- 52/86-TACAN  Receiver,  Intermediate  Main- 
tenance (C-102-3054)  (BNG) 

b.  A- 6A/KA- 6D/A- 6E- Elect r ical  Systems,  Organiza- 
tional Maintenance  (C-602-3766)  (AHL) 

c.  F4B/J-Basic  Aviation  Electrical  System  (C-602-3821) 
(FEGJ 

d.  F4B/J- Advanced  Aviation  Electrical  System 
(C-602-3822)  (FEH). 

3.15  The  graduation  rosters  for  these  four  courses  contained 

the  final  grades  for  all  students  who  completed  these  courses 
from  September  1974  through  November  1976.  Three  samples  were 
developed  for  each  course:  ISD-type,  TI,  and  all  other  students 

who  had  completed  the  same  course.  Then  two  comparisons  for 
each  course  were  made:  ISD-types  were  first  compared  with  the 

TI  students,  and  then  the  ISD-types  were  compared  with  all  other 
students  completing  the  same  course. 

3.16  Cecil  Field  Data.  The  social  security  numbers  of  the 
graduates  in  the  four  cohorts  were  used  to  extract  the  final 
grades  and  course  numbers  from  the  data  base  at  Cecil  Field. 

The  two  AV  cohorts  had  five  ISD-type  and  15  TI  graduate  records. 
The  five  ISD-type  graduates  had  not  received  training  in  the 
same  courses  taken  by  the  TI  graduates;  therefore,  no  perfor- 
mance comparison  could  be  made.  The  two  AE  cohorts  had  21  ISD- 
type  and  11  TI  graduate  records.  One  course  had  records  for 
both  ISD-type  and  TI  students.  However,  only  two  students  of 
each  type  had  taken  the  same  course,  and  no  performance  com- 
parison could  be  made.  Therefore,  Cecil  Field  data  could  not 

be  used  to  provide  a sample  for  NAMTRAGRU  analysis. 
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Statistical  Procedures 


5.17  Each  comparison  group,  i.e.,  ISD-type  versus  TI  and  ISD- 
type  versus  others,  passed  the  F-ratio  test  for  equal  variances, 
and  the  analysis  of  variance  (ANOVA)  test  was  used  to  test  the 
hypotheses.  The  significance  level  of  a = 0.01  was  used  to 
determine  the  results  (see  Appendix  C for  details). 

3.18  Table  5.2  indicates  the  results  of  the  analysis: 

• Column  1 identifies  the  type  of  comparison 
and  the  course  number  (course  titles  are 
identified  in  the  footnotes) 

• Column  2 gives  the  means,  standard  deviations, 
and  sample  sices  of  the  ISD-type  samples 

• Column  5 presents  the  means,  standard  deviations, 
and  the  sample  sizes  of  the  TI  or  "all  other" 
student  samples  taking  that  course 

• Column  4 lists  the  A.N0V.4  F-ratio  statistic 

• Colu:.ins  5 and  6 present  the  F-value  for  the 
level  of  significance  at  a = 0.01  and  a = 0.05, 
and  indicates  whether  the  significance  at  the 

a - 0.01  and  a = 0.05  levels  is  confirmed  or 
negated . 

Results 

3.19  The  analysis  of  the  courses  given  at  NAMTRAGRU,  Oceana, 

consisted  of  29  ISD-type  and  13  TI  graduates.  The  sample 

— ^ These  students  comprise  the  survivors  of  the  initial  large 
cohorts  taken  at  Millington  (see  Paragraph  3.13).  Since  it 
was  not  possible  at  Millington  to  identify  the  NA.MTRAGRU  to 
which  AV  and  AE  Class  graduates  had  been  assigned,  it  was 
not  feasible  to  ensure  a larger  sample  at  NAMTR.4GRU , Oceana, 
by  "seeding"  the  initial  cohorts. 
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TABLE  5.: 

AV  AND  AE  PERFORMANCE  AT  OCE.ANA  N.AMTRAGRU  SCHOOL 


Category 

ISD-Type 

.ANOVA 
F - R a t i 0 

ct  = 0.01 
Level 

a = 0.05 
Level 

I SD- type 
vs 

AV(AT) 

M = 74.77 
S = 1.16 
N = 3 

M = 80.4(5 
S = 5.38 
N = 5 

3.97 

No 

F = 13.74 

No 

F = 5.99 

ISO- type 
vs 

others-- 
AV(AT)  2/ 

M = 74.77 
S = 1.16 
N = 3 

M = 83.44 
S = 5.98 
N = 59 

6.21 

No 

F = 7.08 

Yes 

F = 4.00 

ISO- type 
vs 
TI-- 
AE  3/ 

M = 81.54 
S = 5.92 
N = 14 

M = 77.43 
S = 7.15 
N = 5 

1.60 

No 

F = 8.40 

No 

F = 4.45 

ISO- type 
vs 

others-  - 
AE  3/ 

M = 81.54 
S = 5.92 
N = 14 

M = 80.52 
S = 6.82 
N = 162 

0.29 

No 

F = 6.76 

No 

F = 3.89 

ISO- type 
vs 
TI-- 
AE  4/ 

M = 89.30 
S = 4.92 
N = 7 

M = 33.20 
S = 4.50 
N = 5 

4.80 

No 

F = 10.04 

No 

F = 4.96 

I SD- type 
vs 

others-  - 
AE  4/ 

M = 89.30 
S = 4.92 
N = 7 

M = 89.28 
S = 5.27 
N = 128 

0 

No 

F = 6.81 

■ 

I SD- type 
vs 

TI-- 
AE  5/ 

M = 89.72 
S = 2.25 
N = 5 

M = 84.73 
S = 6.53 
N = 3 

2.65 

No 

F = 13.74 

No 

F = 5.99 

I SD- type 
vs 

others- - 
AE  5/ 

M = 89.72 
S = 2.25 
N = 5 

M = 39.64 
S = 6.03 
N = 103 

0 

No 

F » 6.81 

No 

F = 3.92 

Note:  M is  inean,  S is  standard  deviation,  and  S is  number. 

All  other  students  who  completed  the  course. 

ARN-52/86  TACAN  Receive,  Intermediate  Maintenance  (Course 
No.  C-102-5034)  (BNG). 

— ^ A- 6A/KA- 60/ A- 6E  Electrical  Systems,  Organitational  Maintenance 
(Course  No.  C-602-3766)  (AHL). 

4/ 

F4B/J  Basic  Aviation  Electrical  System  (Course  No.  C-602-3321) 
(FEG3  . 

— F4B/J  Advanced  Aviation  Electrical  System  (Course  No.  C-602- 
3822)  (FEH). 
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>,  sizes  for  each  course  are  considered  to  be  too  small  to  make 

a decision  as  to  the  comparative  quality  of  the  ISD-type  grad- 
uates. However,  the  results  of  the  analysis  are  presented  so 
it  may  be  considered  with  the  data  from  the  other  three  career 
points.  The  results  were: 

a.  Only  three  ISD-type  graduates  (AVs)  took 
the  first  course  shown  in  Table  5.2.  These 
three  students  had  lower  mean  scores  than 
the  TI  students  and  lower  mean  scores  than 
all  other  students  who  completed  the  same 
course . 

1.  The  difference  was  not  significant  at 
the  a = 0.01  level  for  either  compari- 
son . 

2.  The  difference  was  significant  at  the 
a = 0.05  level  for  the  ISD-type  and 
Others  comparison. 

b.  The  results  from  the  analyses  of  the  other 
three  NAMTRAGRU  courses  indicated  the  ISD- 
type  graduates  (.AEs)  had  higher  mean  scores 
than  the  TI  graduates  and  higher  mean 
scores  than  all  other  students  who  completed 
the  same  course. 

1.  The  differences  were  not  significant 
at  the  a = 0.01  level. 

2.  The  differences  were  not  significant 
at  the  a = 0.05  level . 


Conclusions 

3.20  The  conclusions  for  this  portion  of  the  study  were  based 
on  a significance  level  of  a = 0.01.  They  were; 
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• There  was  no  difference  between  the  mean 
final  grades  of  the  ISD-type  and  TI  grad- 
uates taking  the  same  NAMTRAGRU  courses. 

• There  was  no  difference  between  the  mean 
final  grades  of  the  ISD-type  and  all  other 
students  who  completed  the  same  N.AMTRAGRU 
courses . 

PERFORNUNCE  IN  FLEET  .\ND  SHORE  BILLETS  .\ND  ON  THIRD  CLASS 
ADVANCEMENT  EX.AMI NATION 

5.21  The  quality  of  ISD-type  graduates  can  be  evaluated  by 
comparing  their  on-the-job  performance  marks  and  scores  on  the 
Third  Class  Advancement  E.xamination  with  TI  graduates.  When 
class  rosters  of  TI  graduates  were  not  available  at  the  school 
[because  of  records  disposal),  ISD-type  graduates  were  compared 
with  all  A-school  graduates  from  the  same  courses.  If  the  data 
did  not  identify  .A.-school  graduates  then  the  ISD-type  students 
were  compared  with  all  members  of  that  rate  within  the  respec- 
tive rating.  The  three  Class  A courses  selected  for  this  por- 
tion of  the  study  were; 

• Aviation  Electrician's  Mate  (AE) 

• Engineman  (EN)  (nonnuclear  students) 

• Machinists  Mates  (MM)  (nonnuclear  students). 

3.22  The  Aviation  Electrician  (AE)  school  had  records  on  both 
ISD-type  and  TI  graduates,  and  two  cohorts  were  formed.  The 
school  records  included  the  BTB  scores  so  that  a comparison 

of  the  quality  of  students  selected  for  the  AE  course  could 
also  be  made. 
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3.23  The  Engineman  (EX)  and  Machinists  Mates  (MM)  schools  only 
had  ISD-type  records  available.  Therefore,  the  ISD-type  grad- 
uates' third  class  examination  scores  were  compared  with  all 
A-school  graduates  from  the  same  school.  The  data  for  the  on- 
the-job  performance  marks  did  not  identify  the  A-school  grad- 
uates, so  the  ISD-type  graduates  were  compared  with  all  E-3s 
within  the  respective  rating. 

Hypotheses 

3.24  The  hypotheses  tested  for  the  on-the-job  performance  marks 
analysis  were: 

• There  is  no  difference  between  the  mean 
on-the-job  performance  marks  of  the  ISD- 
type  and  TI  graduates  from  the  AE  school 

• There  is  no  difference  between  the  mean  BTE 
scores  of  the  ISD-type  and  TI  graduates 
selected  for  the  AE  school 

• There  is  no  difference  between  the  mean  on- 
the-job  performance  marks  of  the  EX,  ISD- 
type  graduates,  and  all  EX  E-3s  recommended 
for  promotion  to  Third  Class 

• There  is  no  difference  between  the  mean  on- 
the-job  performance  marks  of  the  MM,  ISD- 
type  graduate,  and  all  MM  E-3s  recommended 
for  promotion  to  Third  Class. 

3.25  The  hypotheses  tested  for  performance  on  the  Third  Class 
Advancement  Examinations  were: 
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• There  is  no  difference  between  the  mean  scores 
on  the  Third  Class  Advancement  Examination  of 
the  ISD-type  and  TI  graduates  of  the  AE  course. 

• There  is  no  difference  between  the  average  per- 
formances on  the  Third  Class  Advancement  Exami- 
nation of  the  ISD-type  and  all  EN , A-school 
graduates  from  the  EN  course. 

• There  is  no  difference  between  the  average  per- 
formances on  the  Third  Class  .Advancement  Exami- 
nation of  the  ISD-type  and  all  MM,  A-school 
graduates  from  the  MM  course. 

Data  and  Data  Collection 


3.26  The  Naval  Education  and  Training  Information  Systems 
Activity  (NET ISA)  data  base  at  Pensacola,  Florida,  contained 
data  on  the  Third  Class  .Advancement  Examination  results.  Each 
graduate's  social  security  number  was  used  to  extract  the  fol- 
lowing data: 

• On-the-job  performance  mark 

• Raw  score  on  the  examination 

• Standardized  score  on  the  examination. 

The  Systems  .Advancement  Evaluation  and  Analysis  Division,  Naval 
Education  and  Training  Program,  Ellyson  Field,  Pensacola,  Florida, 
furnished  the  averages,  standard  deviations,  and  sample  sizes 


— When  personnel  are  recommended  to  take  the  Third  Class  .Ad- 
vancement Examination,  their  command  must  submit  an  average 
of  the  candidate's  last  several  enlisted  performance  eval- 
uations. This  average,  or  on-the-job  enlisted  performance 
mark,  provides  the  best  statistical  measure  of  performance. 
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of  on-the-job  performance  marks  for  candidates  taking  each  exami- 
nation, by  rate  and  rating.  They  also  furnished  the  distribu- 
tion of  standard  scores  for  A-school  candidates  by  rate  and 
rating  for  each  examination. 

3.27  AE.  The  original  cohorts  of  about  500  each  had  Marine, 
foreign,  and  reserve  students.  When  these  students  were  deleted 
from  the  cohorts,  the  TI  cohort  contained  392  graduates,  and 

the  ISD-type  contained  346  graduates.  Both  AE  cohorts  graduated 
in  early  1975,  and  data  from  the  August  1975,  February  1976, 
and  August  1976  test  periods  provided  information  on  45%  of 
each  cohort  or  153  ISD-types  and  175  TI  graduates  for  compari- 
son . 

3.28  EN  and  MM.  The  EN  and  MM  ISD-type  graduates  graduated 
in  early  1974.  Each  cohort  had  several  graduates  take  the  ex- 
amination in  August  1974,  but  comparison  data  were  not  available 
from  Ellyson  Field  for  that  test  period.  The  NETISA  data  base 
was  missing  data  on  the  February  1975  test  period.  Complete 
data  for  comparison  were  available  from  both  sources  for  August 
1975,  February  1976,  and  August  1976  examinations.  These  three 
test  periods  contained  249  EN , ISD-type  graduates  (50%  of  the 
initial  cohort)  and  201  MM  ISD-type  graduates  (40%  of  the  or- 
iginal cohort ) . 

3.29  Third  Class  Advancement  Examination.  The  raw  and  standard 
scores  were  used  to  assess  performance  on  the  Third  Class  Ad- 
vancement Examination.  The  raw  score  indicated  how  well  the 
candidates  did  on  the  examination.  When  the  raw  scores  were 
standardized,  the  performance  of  all  of  the  other  personnel 

in  that  rating  (taking  the  same  examination)  influenced  the 
computation  of  each  candidate's  standard  score.  Standard  scores 
were  used  to  compare  students  on  the  same  examination,  and  raw 
scores  were  used  to  compare  students  across  examinations. 
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3.50  .A.E . The  AE  cohorts  passed  the  test  for  equal  variances. 

The  .ANOV.-\  statistical  test  was  used  to  compare  both  cohorts 

for  on-the-job  performance  marks  and  for  Third  Class  .Advancement 
Examination  scores. 

5.51  EN  and  MM.  The  EN  and  NLM  cohorts  failed  the  equal  vari- 
ance test  for  both  performance  marks  and  examination  test  scores. 
Therefore,  the  two-tailed  t-test  was  used  to  compare  the  ISD- 
type  graduates  with  all  E-5s  who  had  on-the-job  performance 
marks  submitted  for  that  test  period  as  well  as  with  .A-school 
graduates  who  took  the  Th’rd  Class  Advancement  Examination. 

3.52  Tables . Tables  3.3  through  5.8  illustrate  the  results 
of  the  analysis.  They  were  formatted  as  follows: 

• Column  1 gives  the  date  of  examination  and 
indicates  whether  the  table  contains  on-the- 
job  performance  marks  or  standard  scores  for 
third  class  examination  results. 

• Column  2 shows  the  means,  standard  deviations, 
and  sample  sizes  for  the  ISD-type  graduates. 

• Column  3 gives  the  means,  standard  deviations, 
and  sample  sizes  for  the  comparison  groups. 

The  heading  at  the  top  of  the  column  indicates 
the  type  of  comparison  group;  TI  , A-school 
graduates  or  all  E-3s  within  the  respective 

r ate . 
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TABLE  3.5 

AE  ON-THE-JOB  PERFOR.M.ANCE  NLARKS 


Category 

ISD-Type 

TI 

■ANOVA 
F-Rat  io 

a = 0.01 
Level 

QBQIfQ 

Performance 
marks , 
August  1975 

M = 5.5  2 
S = 0.14 
N = 15 

M = 5.5  5 
S = 0.16 
N = 19 

0.46 

No 

F = 7.50 

No 

F = 4.15 

Performance 
marks  , 

February  1976 

M = 5.59 
S = 0.15 
N = 5 

•M  = 5.5  9 
S = 0.18 
N = 57 

0.0015 

No 

F = 7.08 

No 

F = 4.00 

Performance 
marks, 
August  1976 

M = 5.5  9 
S = 0.19 
N = 155 

M =5.54 
S = 0.20 
N = 99 

5 . 59 

No 

F = 6.70 

No 

F = 5.86 

Performance 
marks  over- 
all cycles 

M = 5.5  8 
S = 0.18 
N = 155 

M = 5.56 
S = 0.19 
N = 175 

1.47 

No 

F = 6.70 

No 

F = 3.86 

BTB 

scores* 

M = 169.97 
S = 16.26 
N = 15  3 

M = 169.62 
S = 14.57 
N = 175 

0.04 

No 

F = 6.70 

No 

F = 3.86 

Note:  M is  mean,  S is  standard  deviation,  and  N is  number. 

* BTB  score  = ARI  + 2 ETST . 
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TABLE  3.4 

EN  ON-THE-JOB  PERFORNLANCE  MARKS 


Category 

ISD-Type 

All  EN 
E-  3s 

IWHWI 

a = 0.05 
Level 

Performance 
marks , 
■August  1975 

M = 3.57 
S = 0.19 
N = 57 

M = 3.5  3 
S = 0.20 
N = 480 

1.42 

No 

t = 2.65 

No 

t = 2.00 

Performance 
marks  , 

February  1976 

M = 3.5  8 
S = 0.20 
N = 151 

M = 3.5  5 
S = 0.18 
N = 651 

1 . 80 

No 

t = 2.60 

No 

t = 1,97 

Performance 
marks , 
August  1976 

M = 3.50 
S = 0.22 
N = 41 

M = 3.5  5 
S = 0.19 
N = 519 

1.51 

No 

t = 2.67 

No 

t = 2.01 

Performance 
marks  over 
all  cycles 

M = 3.56 
S = 0.20 
N = 249 

M = 3.5  4 
S = 0.19 
N = 1,650 

1.48 

No 

t = 2.53 

No 

t = 1.96 

Note;  M is  mean,  S is  standard  deviation,  and  N is  number. 
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NLM  ON-THE-JOB  PERFORM.ANCE  MARKS 


Category 

— 

ISD-Type 

All  MM 
E-  5s 

mBm 

a = 0.01 
Level 

a = 0.05 
Level 

Performance 
marks  , 
August  1975 

M = 5.5  2 
S = 0.21 
N = 6 2 

M = 5.52 
S = 0.20 
N = 750 

0.23 

No 

t = 2.64 

No 

t = 1.99 

Performance 
marks , 

February  1976 

M = 5.55 
S = 0.54 
N = 110 

M = 5.5  4 
S = 0.20 
N = 8 20 

0.25 

No 

t = 2.60 

No 

t = 1.97 

Performance 
marks  , 
.August  1976 

M = 5.5  5 
S = 0.18 
N = 29 

M = 5.56 
S = 0.20 
N = 912 

O.Sl 

No 

t = 2.76 

No 

t = 2.05 

Performance 
marks  over 
all  cycles 

M = 5.5  5 
S = 0.29 
N = 201 

M = 5.5  4 
S = 0.20 
N = 2,462 

0.50 

No 

t = 2.60 

No 

t = 1.97 

Note:  M is  mean,  S is  standard  de 
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TABLE  3.6 

AE  PERFORMA.N’CE  ON  THIRD  CLASS  ADVANCEMENT  EX.AMINATION 


Category 

I SD- Type 

TI 

ANOVA 
F-Rat  io 

a = 0.01 
Level 

a = 0.05 
Level 

Standard 
scores , 
August  1975 

M = 55.47 
S = 8.51 
N = 15 

M = 52.63 
S = 9.25 
N = 19 

0.84 

No 

F = 7.50 

No 

F = 4.15 

Standard 
scores  , 
February  1976 

M = 52.60 
S = 10.95 
N = 5 

M = 50.25 
S = 6.84 
N = 57 

2.55 

No 

F = 7.08 

No 

F = 4.00 

Standard 
scores , 
August  1976 

M = 52.08 
S = 8.54 
N = 135 

M = 51.14 
S = 9.20 
N = 99 

0.66 

No 

F = 6.70 

No 

F = 5.86 

Number 

right 

overall 

M = 78.47 
S = 14.26 
N = 153 

M = 76.79 
S = 14.45 
N = 1'5 

1 . 14 

1 

No 

F = 6.70 

No 

F = 3.86 

Note:  M is  mean,  S is  standard  de 
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TABLE  3.7 

EN  PERFORNLANCE  ON  THIRD  CLASS  ADVANCEMENT  EX.VMINATION 


Category 

ISD-Type 

-A-School 

Graduates 

ww 

a = 0.01 
Level 

mm 

Standard 
scores , 
August  1975 

M = 52.26 
S = 9.99 
N = 57 

M = 55.54 
S = 8.96 
N = 249 

2 . 14 

No 

t = 2.65 

Yes 

t = 2.00 

Standard 
scores , 
February  1976 

M = 55.90 
S = 7.97 
N = 151 

M = 55.44 
S = 8.88 
N = 54  2 

0.57 

No 

t = 2.60 

No 

t = 1.97 

Standard 
scores , 
August  1976 

M = 55.58 
S = 10.21 
N = 42 

M = 55.55 
S = 9.08 
N = 319 

1 . 25 

No 

t = 2.67 

No 

t = 2.00 

Note:  M is  mean,  S is  standard  deviation,  and  N is  number. 


52 


1 


TABLE  3.8 

■MM  PERFOR.VIANCE  ON  THIRD  CLASS  ADVANCEMENT  E.X.AMINATION 


Category 

I SD- Type 

.A-  School 
Graduates 

saa 

Baa 

Standard 

M = 51.37 

M = 54.70 

2.87 

Yes 

Yes 

scores  , 

S = 8.25 

S = 10.21 

t = 2.64 

t = 1.99 

.August  1975 

N = 6 2 

N = 418 

Standard 

M = 53.80 

M = 53.51 

0.38 

No 

No 

scores  , 

S = 6.93 

S = 9.11 

t = 2.60 

t = 1.97 

February  1976 

N = 110 

N = 472 

Standard 

M = 56.90 

M = 53.67 

2.31 

No 

Yes 

scores  , 

S = 7.16 

S = 9.55 

t = 2.75 

t = 2.04 

■August  1976 

N = 29 

--  - 

N = 512 

1 

Note:  M is  mean,  S is  standard  deviation,  and  N is  number. 
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• Column  4 lists  the  ANOVA  F-ratio  or  t- 
statistic. 

• Columns  5 and  6 present  the  appropriate  F- 
or  t-values  for  the  level  of  significance  at 
ct  = 0.01  and  a = O.OS  and  whether  the  sig- 
nificance at  the  a = 0.01  and  a = 0.05  levels 
is  confirmed  or  negated. 

For  more  details  on  the  statistical  methodology,  see  .Appendix  C. 

Results  of  On-the-Job  Performance  Marks  Analy s i s 


5.33  AE . Table  3.3  illustrates  the  results  of  the  AE  ISD-type 
and  TI  graduates'  on-the-job  performance  marks.  The  results 
were ; 

a.  TI  graduates  had  higher  on-the-job 
performance  marks  than  the  ISD-type 
graduates  for  the  August  1975  test 
period,  but  the  difference  was  not 
significant 

b.  The  ISD-type  and  TI  graduates  had  the 
same  on-the-job  performance  marks  sub- 
mitted for  the  February  1976  test 
period 


Q / 

— The  19  TI  graduates  had  an  average  of  6 months  of  service 
from  graduation  until  August  1975.  The  15  ISD-type  grad- 
uates had  an  average  of  2 months  and  9 days  of  service  from 
graduation  until  August  1975;  this  may  account  for  the  lower 
performance  marks  for  the  ISD-type  graduates. 
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The  ISD-type  graduates  had  higher  on- 
the-job  performance  marks  submitted  for 
the  August  1976  test  period 

When  the  ISD-type  graduates  were  compared 
with  all  TI  graduates  over  all  three  test 
periods,  the  ISD-type  graduates  had  higher 
on-the-job  performance  marks 

The  differences  on  the  four  comparisons 
were  not  significant 

The  BTB  scores  of  the  ISD-type  graduates  were 
higher,  but  the  difference  was  not  significant. 

3.34  EN.  Table  3.4  illustrates  the  results  of  the  EN  ISD-type 
graduates  and  all  EN  E-3s  who  had  on-the-job  performance  marks 
submitted.  The  results  were: 

a.  The  ISD-type  graduates  had  higher  mean  on- 
the-job  performance  marks  submitted  for 
the  first  two  test  periods  • 

b.  The  TI  graduates  had  higher  mean  on-the- 
job  performance  marks  submitted  for  the 
last  test  period 

c.  When  the  ISD-type  graduates  were  compared 
with  all  the  EN  E-3s  over  all  three  test 
periods,  the  ISD-type  graduates  had  higher 
on-the-job  performance 

d.  None  of  the  four  comparisons  was  significantly 
different. 

3.35  Table  3.5  illustrates  the  results  of  the  MM  ISD-type 
graduates  and  all  MM  E-3s  who  had  on-the-job  performance  marks 
submitted.  The  results  were: 
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I a.  The  ISD-type  graduates  and  all  the  NIM  E-5s 

I had  the  same  mean  scores  submitted  for  the 

August  1975  test  period 

b.  The  ISD-type  graduate  had  lower  on-the-job 
performance  marks  submitted  for  the  last  two 
test  periods 

c.  When  the  on-the-job  performance  marks  were 
compared  over  all  three  tests,  the  ISD-type 
graduates  had  lower  mean  scores  than  all  the 
MM  E-5s 

d.  None  of  the  four  comparisons  was  significantly 
different . 

Conclusions  for  On-the-Job  Performance  Marks 


5.36  The  conclusions  for  this  portion  of  the  study  were  based 
on  a significance  level  of  a = 0.01.  They  were; 

• There  was  no  difference  between  the  mean  on- 
the-job  performance  marks  of  the  ISD-type  and 
TI  graduates  from  the  AE  school 

• There  was  no  difference  between  the  mean  BTB 
scores  of  the  ISD-type  and  TI  graduates 
selected  for  the  AE  school 

• There  was  no  difference  between  the  mean  on- 
the-job  performance  marks  of  the  EN  ISD-type 
graduates  and  all  EN  E-5s  recommended  for 
promotion  to  Third  Class 

• There  was  no  difference  between  the  mean  on-the- 
job  performance  marks  of  the  MM  ISD-type  gradu- 
ates and  all  MM  E-5s  recommended  for  promotion 
to  Third  Class . 


/ 
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Results  of  Third  Class  Advancement  Examination  Analysis 

5.37  AE . Table  5.6  illustrates  the  results  of  the  AE  ISD-type 
and  TI  graduates'  performances  on  the  Third  Class  Advancement 
Exami nat i on : 

a.  The  ISD-type  graduates  had  higher  mean  stan- 
dard scores  on  each  examination  than  the  TI 
graduates 

b.  The  ISD-type  graduates  had  higher  mean  raw 
scores  for  all  examinations  than  the  TI 
graduates 

c.  The  ISD-type  graduates  had  higher  mean  scores 
on  all  four  comparisons,  but  none  of  the  dif- 
ferences was  significant  at  either  level. 

5.58  EN . Table  3.7  illiotrates  the  results  of  the  performances 
of  the  ISD-type  and  A-school  graduates  from  the  EN  courses  on 
the  Third  Class  .\dvancement  Examination: 

a.  The  A-school  graduates  did  better  than  the 
ISD-type  graduates  on  the  August  1975  test: 

1.  The  difference  was  not  significant  at 
the  a = 0.01  level 

2.  The  difference  was  significant  at  the 
a = 0.05  level 

b.  The  ISD-type  graduates  had  higher  mean  scores 
on  the  second  and  third  examinations: 

1.  The  differences  were  not  significant 
at  ei ther  level  . 
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5.59  MM.  Table  5.8  illustrates  the  results  of  the  comparison 
between  ISD-type  and  all  .<\-school  graduates  on  the  Third  Class 
.Advancement  E.xami nat  i on ; 

a.  The  A-school  graduates  had  higher  test 
scores  on  the  .August  1975  examination 
(the  difference  was  significant  at  both 
levels ) 

b.  The  ISD-type  graduates  had  higher  test 
scores  on  the  February  1976  examination 
(the  difference  was  not  significant  at 
either  level) 

c.  The  ISD-type  graduates  had  higher  test 
scores  on  the  .August  1976  examination 
(the  difference  was  not  significant  at 
the  a = 0.01  level;  the  difference  was 
significant  at  the  a = 0.05  level). 

Conclusions  for  Third  Cl  a ss  .Advancement  Examination 


5.40  The  conclusions  for  this  portion  of  the  study  were  based 
on  a significance  level  of  a = 0.01.  They  were: 

• There  was  no  difference  between  mean  scores 
on  the  Third  Class  Advancement  Examination 
of  the  ISD-type  and  TI  graduates  of  the 

AE  course 

• There  was  no  difference  between  the  mean 
scores  on  the  Third  Class  .Advancement 
Examination  of  the  ISD-type  and  all  EN 
E-5  graduates  of  the  E.N  .A-school 
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• There  was  no  difference  between  the  scores 
on  the  Third  Class  Advancement  Examination 
of  the  ISD-type  and  all  MM  E-5s  who  gradu- 
ated from  the  MM  A- school. 
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IV.  FINDINGS  AND  CONCLUSIONS 


4.1  This  section  presents  the  findings  and  conclusions  of 
the  research  effort.  The  research,  primary  data  gathering, 
and  analysis  were  directed  towards  graduate  quality.  Two  types 
of  comparison  were  made:  (a)  comparison  of  graduate  quality 

from  ISD-type  courses  with  graduate  quality  from  the  same  courses 
in  a TI  configuration  and  (b)  curriculum  review  comparison  of 
the  curriculum,  AOB,  attrition,  and  costs  of  the  courses  se- 
lected for  research. 

FINDINGS 

Curriculum  Review 


4,2  Curriculum.  The  following  were  the  major  findings  from 
the  ISD-type  and  TI  curricula  comparison: 

a.  The  .AV  .A  course  ISD-type  curriculum  contained 
all  the  major  units  that  were  in  the  TI  cur- 
riculum, but  the  course  hours  were  shortened 

to  485.5  contact-hours  from  511.7  contact -hours . 

b.  The  most  noticeable  difference  in  the  AE  .A 
course  ISD-type  curriculum  was  the  inclusion 
of  a specific  troubleshooting  unit  rather  than 
the  treatment  of  troubleshooting  as  a non- 
specific element  of  other  units. 
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c.  N’o  theory  units  were  specified  in  the  ISD-type 
Basic  Propulsion  Engineering  (PE)  Course,  and 
there  was  a considerable  change  in  the  equip- 
ment included  in  the  two  courses. 

4.3  AOB . Major  findings  from  the  TI  and  ISD-type  course  AOB 
contrast  are  as  follows; 

a.  There  were  significant  reductions  in  AOB  in 
the  EN  and  MM  A courses. 

b.  No  change  in  .AOB  in  the  AE  .A  course  occurred. 

The  course  was  not  self-paced  in  either  the 
ISD-type  or  TI  configuration,  and  the  course 
length  remained  at  11  wk. 

4.4  .Attrition.  The  comparison  of  TI  and  ISD-type  courses' 
attrition  provided  the  following  data: 

a.  Both  academic  and  total  attrition  were  worse 
in  the  .AV  .A  and  basic  PE  ISD-type  courses  than 
in  the  TI  configuration. 

b.  AE  .A  course  ISD-type  academic  attrition  was 
the  same,  and  total  attrition  was  better  than 
in  the  TI  configuration. 

c.  The  range  of  academic  attrition  for  the  three 
courses  was  0.9%  to  5.5%,  and  total  attrition 
had  a range  of  1.5%  to  11.2%. 

4.5  Cost . The  following  were  the  major  findings  from  the 
TI  and  ISD-type  course  per - capi ta- cost  - to- tr a in  contrast: 
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a.  The  per  capita  cost  to  train  for  all  four  of 
the  courses  examined  was  higher  for  the  ISD- 
type  configuration  than  the  TI  configuration, 
but  three  of  the  four  course  costs  were  well 
within  the  range  attributable  to  inflationary 
influences . 

b.  The  fourth  ISD-type  course  (EN  A)  cost  was 
influenced  by  a large  construction  investment 
expense . 

c.  Given  present  cost  accounting  procedures, 

u major  study  that  is  cost  oriented  would  be 
ntcessary  to  analyte  the  cost  impact  of  con- 
verting a course  to  ISD  from  TI. 

Graduate  Quality 


4.6  The  major  findings  from  the  quantitive  analysis  of  Section 
III  are  described  in  the  following  paragraphs.  They  are  grouped 
first  by  each  ISD-type  course  studied  and  then  by  career  points 
as  defined  in  Paragraph  3.1.  Table  4.1  summarizes  the  findings. 

Avionics  Technician  (AV)  Course 


4.7  The  AV  ISD-type  graduates  were  evaluated  at  the  Advanced 
First  Term  ,‘\vionics  (.\FTA)  Course  (a  follow-on  B-school).  Each 
AFTA  student  was  graded  in  four  test  areas,  and  a final  course 
grade  was  assigned.  The  findings  were  as  follows: 
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NAMTRAGHU  courses  (see  footnotes  of  Table  3.2  on  p.  40). 
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a.  The  ISD-type  graduates'  grades  in  the  two 
test  areas  of  practical  work,  and  performance 
tests  were  significantly  higher  than  the 
TI  graduates'  grades. 

b.  The  ISD-type  graduates'  grades  in  the 
areas  of  written  tests  and  comprehensive 
examinations  were  higher  than  the  TI  gradu- 
ates' grades. 

c.  The  final  course  grade  is  a weighted  com- 
posite of  the  four  test  grades  just  dis- 
cussed. The  ISD-type  had  higher  final  grades, 
but  they  are  not  significantly  higher. 

d.  The  ISD-type  graduates  had  higher  prerequi- 
site Basic  Test  Battery  (BTB)  scores  used  for 
selection  to  the  AV  course,  but  they  were  not 
significantly  higher. 

Aviation  Electrician's  Mate  (AE)  Course 


4.8  The  AE  ISD-type  graduates  were  evaluated  at  three  career 
points:  Naval  Air  Maintenance  Training  Group  (NAMTRAGRU)  courses 

at  Oceana,  on-the-job  performance  marks  submitted  for  promotion 
to  Third  Class  and  the  scores  on  the  Third  Class  .Advancement 
Examination.  ISD-type  graduates  were  compared  to  TI  graduates 
at  all  three  career  points.  Additionally,  ISD-type  graduates 
were  compared  to  all  students  taking  the  same  course  at  the 
NAMTRAGRU.  The  findings  were  as  follows: 


- The  word  significant  used  in  this  context  indicates  that 
the  difference  in  the  scores  was  statistically  different 
at  the  or  = 0.01  level  of  significance. 
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a.  The  ISD-type  graduates  had  higher  final 
grades  than  the  TI  graduates  and  all  students 
taking  the  three  courses  at  the  NAMTRAGRU . 

The  differences  were  not  significant. 

b.  The  ISD-type  graduates'  on-the-job  perfor- 
mance marks  submitted  for  the  first  promotion 
period  (August  1975)  were  lower  than  the  TI 
graduates'.  For  the  second  period  (February 
1976)  they  had  the  same  marks,  and  for  the 
third  period  (August  1976)  the  ISD-type  grad- 
uates had  higher  marks.  When  all  the  ISD-type 
graduates'  on-the-job  performance  marks  were 
compared  to  all  the  TI  graduates'  marks,  the 
former  had  higher  scores.  The  differences 
were  not  significant. 

c.  The  ISD-type  graduates  had  higher  prerequisite 
BTB  scores,  but  they  were  not  significantly 
higher . 

d.  The  ISD-type  graduates  had  higher  Third  Class 
Advancement  Examination  scores  than  the  TI 
graduates  on  all  three  examinations.  The 
scores  were  not  significantly  higher. 


PE  School 


4.9  The  Engineman  (EN)  and  Machinists  Mates  (MM)  ISD-type 
graduates  were  evaluated  at  two  career  points -- on- the- job  per- 
formance marks  submitted  for  Third  Class  promotion  and  their 
scores  on  the  Third  Class  Advancement  Examination.  A TI  cohort 
could  not  be  constructed  from  records  available  at  the  PE  school. 
Therefore,  performance  marks  were  compared  with  all  candidates 
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taking  the  same  Third  Class  Advancement  Examination.  The  ISD- 
type  graduates'  scores  on  the  examination  were  compared  with 
the  scores  of  all  A-school  graduates  taking  the  same  examination 
with  the  same  rates  and  ratings.  The  findings  were  as  follows: 

a.  The  EN  ISD-type  graduate  had  higher  on-the- 
job  performance  marks  submitted  for  the  first 
two  promotion  periods  (August  1975  and  Febru- 
ary 1976)  and  lower  marks  for  the  last  period 
(August  1976).  When  all  the  ISD-type  gradu- 
ates were  compared  with  all  EN  candidates  for 
Third  Class  promotion,  the  ISD-type  graduates 
had  higher  on-the-job  performance  marks.  The 
differences  were  not  significant. 

b.  The  EN  ISD-type  graduates  had  lower  scores 
than  the  A-school  graduates  on  the  first  Third 
Class  .Advancement  Examination  and  higher  scores 
on  the  last  two  examinations.  The  differences 
were  not  significant. 

c.  The  MM  ISD-type  graduates  and  all  MM  candidates 
for  Third  Class  promotion  had  equal  on-the- 

job  performance  marks  submitted  for  the  first 
period  (August  1975),  and  the  ISD-type  gradu- 
ates had  lower  on-the-job  performance  marks  sub- 
mitted for  the  last  two  periods  (February  and 
August  1976).  When  the  comparison  was  made 
over  the  three  test  periods,  the  ISD-type  grad- 
uates had  lower  performance  marks.  The  differences 
were  not  significant. 
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d.  The  MM  ISD-type  graduates  had  significantly 
lower  scores  on  the  first  Third  Class  Advance- 
ment Examination  than  the  A-school  graduates. 
The  ISD-type  graduates  had  higher  scores  on 
the  last  two  Third  Class  Advancement  Examina- 
tions. 

e.  Comparison  between  graduates'  test  scores  from 
the  three  different  test  periods  on  the  Third 
Class  Advancement  Examination  could  not  be 
made.  The  raw  scores  indicated  the  tests  were 
significantly  different  (for  both  the  ENs  and 
MMs)  . 

Career  Points 


4.10  The  major  findings  at  each  career  point  are  as  follows: 

a.  The  ISD-type  graduates  performed  better  in  the 
AFTA  course  and  NAMTRAGRU  courses.  The  perfor- 
mances were  not  significantly  better. 

b.  The  on-the-job  performance  marks  were  mixed 
for  each  rating,  with  the  AE  and  E.N  ISD-type 
graduates  having  better  on-the-job  performance 
marks  and  the  MM  having  lower  performance  marks 
than  their  non-ISD-type  cohorts.  The  ISD-type 
graduates'  on-the-job  performance  marks  were 
not  significantly  different  than  the  performance 
marks  of  the  other  candidates  for  promotion  to 
Third  Class. 
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c.  The  ISD-type  graduates'  scores  on  the  Third 
Class  Advancement  Examination  indicated  mixed 
results.  The  only  comparison  that  was  signifi- 
cant was  the  NIMs ' scores  on  the  first  test, 
where  the  ISD-type  graduates  had  significantly 
lower  scores  than  the  A-school  graduates.  The 
ISD-type  graduates  did,  however,  have  higher 
scores  on  seven  of  nine  tests,  but  eight  of 
the  nine  comparisons  indicated  there  was  no 
significant  difference  between  ISD-type  gradu- 
ates' scores  and  the  A-school  graduates'  scores. 

CONCLUSIONS 

4.11  The  conclusions  in  the  following  paragraphs  must  be  con- 
sidered inferential  in  their  application  to  ISD  courses  since 
research  had  to  be  restricted  to  ISD-type  courses. 

Curriculum  Review 


4.12  The  following  conclusions  were  made  concerning  ISD-type 
and  TI  course  curricula: 

a.  ISD-type  courses  contained  less  specifically 
identified  theory  than  did  TI  courses. 

b.  Changing  to  an  ISD-type  configuration  did 
imply  a high  probability  of  reducing  AOB. 

c.  Academic  and  total  attrition  may  get  better 
or  worse  after  an  ISD-type  conversion.  Be- 
cause self-pacing,  student  quality,  and 
quality  of  course  after  conversion  are  also 
significant  factors  in  determining  attrition, 
the  changes  in  attrition  could  not  be  directly 
linked  solely  to  the  type  of  course. 
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No  costing  conclusions  could  be  made  regarding  the  ISD-type 
course  compared  to  the  TI  configuration. 

Graduate  Quality 

4.13  The  graduate  quality  from  ISD-type  courses  is  equivalent 
to  TI  course  graduates  in  terms  of  performance  in  follow-on 
training,  on-the-job  performance,  and  in  Third  Class  Advance- 
ment Examination  results. 
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APPENDIX  A 
PERSONNEL  CONTACTED 


Name 

Title 

Organization 

Telephone 

Mr. 

D. 

B.  Adams 

Branch  Head  of 
the  Attack  Train- 
ing Branch 

NavAi  r 

222-0947 

Mr. 

P. 

J.  Baranowski 

Director,  System 
Statistical  Analy- 
sis Branch 

Naval  Education 
and  Training 
Program  Devel- 
opment Center 

922-1681 

CDR 

J. 

Bash 

Engineering  Systems 
Training  Branch 

CNTT,  N-33 

966-5894 

Mr. 

Bowell 

Educational  Spe- 
cialist, AE  "A" 
Course 

NATTC 

966-5697 

CDR 

F. 

S.  Bownan 

Pers  55 

CNP 

224-1953 

LCDR  0 

. D.  Brown 

N.AMTRA 

CNTT,  N-422 

966-5984 

CDR 

W. 

Bryant 

NETISSA  Liaison 

CNET 

922-4537 

Mr. 

S. 

Car sen 

CMI  Training  Spe-  • 
cialist 

CNTT,  01531 

966-5375 

Mr. 

L. 

Chambers 

CMI  Management 
Branch 
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APPENDIX  C 

STATISTICAL  METHODOLOGY 


C.l  This  study  required  that  decisions  about  populations  be 
based  on  sample  information.  Such  determinations  are  called 
statistical  decisions.  For  example,  one  may  wish  to  decide  on 
the  basis  of  sample  data  whether  a certain  educational  process, 
vie.,  traditional  instruction,  is  better  than  another,  vie., 
instructional  systems  development  i^ISD)  type.  To  make  statis- 
tical decisions,  the  following  steps  were  followed: 

• Define  a statistical  hypothesis 

• Select  a level  of  significance  for  testing 

• Determine  the  appropriate  statistical  test 

• Test  the  hypothesis 

• State  the  results. 

STATISTICAL  HYPOTHESES 

C.2  In  attempting  to  reach  decisions,  assumptions  or  infer- 
ences about  the  populations  must  be  made.  Such  assumptions, 
which  may  or  may  not  be  true,  are  called  statistical  hypotheses 
and  in  general  are  statements  about  the  probability  distribu- 
tions of  the  populations. 

C.3  In  many  instances  statistical  hypotheses  are  formulated 
for  the  sole  purpose  of  rejecting  or  nullifying  them.  For 
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example,  to  decide  whether  one  educational  process  is  better 
than  another,  the  hypothesis  is  formulated  that  there  is  no 
difference  between  the  graduates  from  the  two  processes  [i.e., 
any  observed  differences  are  merely  due  to  fluctuation  in  sampl- 
ing from  the  same  population).  Such  hypotheses  are  called  null 
hypotheses . 

C.4  Any  hypothesis  that  differs  from  a given  hypothesis  is 
called  an  alternative  hypothesis.  For  example,  if  one  hypothe- 
sis asserts  "there  is  no  difference  between  TI  and  ISD-type 
graduates,"  the  alternative  hypothesis  is,  "there  is  a differ- 
ence between  TI  and  ISD-type  graudates." 

Tests  of  Hypotheses  and  Significance 

C.5  Given  that  a particular  hypothesis  is  true,  if  results 
observed  in  a random  sample  differ  markedly  from  those  expected 
on  the  basis  of  pure  chance  using  sample  theory,  it  vvould  indi- 
cate that  the  observed  differences  are  significant  and  that  the 
hypothesis  should  be  rejected  (or  at  l.east  not  accepted  on  the 
basis  of  the  evidence  obtained) . 

C.6  Statistical  procedures  that  enable  the  analysis  to  accept 
or  reject  hypotheses  or  to  determine  whether  observed  samples 
differ  significantly  from  expected  results  are  called  tests  of 
hypotheses . 

Type  I and  Type  II  Errors 

C.7  If  the  hypothesis  is  rejected  when  it  should  have  been 
accepted,  then  a Type  I error  has  been  made.  If,  on  the  other 
hand,  the  hypothesis  is  accepted  when  it  should  have  been  re- 
jected, a Type  II  error  has  been  made.  In  either  case  a wrong 
decision  or  error  in  judgment  has  occurred. 
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C.8  For  any  test  of  hypothesis  to  be  valid,  it  must  be  de- 
signed to  minimize  the  error  of  decision.  This  is  not  a simple 
matter  since,  for  a given  sample  size,  an  attempt  to  decrease 
one  type  of  error  is  generally  accompanied  by  an  increase  in 
the  other  type  of  error.  The  only  way  to  reduce  both  types  of 
error  is  to  increase  the  sample  size. 

LEVEL  OF  SIGNIFIC.VNCE 

C.9  In  testing  a given  hypothesis,  the  maximum  probability 
with  which  one  is  willing  to  risk  a Type  I error  is  called  the 
level  of  significance  of  the  test.  This  probability,  denoted 
by  a,  is  generally  specified  before  any  samples  are  drawn,  so 
the  results  obtained  will  not  influence  the  choice. 

C.IO  In  practice  a level  of  significance  of  0.05  or  0.01  is 
used  for  testing,  although  other  values  are  sometimes  used.  If, 
for  exa.ziple,  a 0.05  (or  51)  level  of  significance  is  chosen  in 
designing  a test  of  hypothesis,  then  there  are  about  5 chances 
in  100  that  the  hypothesis  would  be  rejected  when  it  should  be 
accepted,  i.e.,  one  is  951  confident  that  the  right  decision 
is  made.  In  such  cases  it  is  said  that  the  hypothesis  has  been 
rejected  at  a 0.05  level  of  s igni f icance , which  means  that  it 
could  be  wrong  with  a probability  of  0.05. 

C.ll  The  sample  sizes  used  in  this  study  were  very  large  when- 
ever sufficient  data  were  available.  .■ill  tests  of  hypotheses 
were  null  hypotheses,  e.g.,  "there  is  no  difference  between  the 
two  comparison  groups.”  A level  of  significance  of  a = 0 . i 
was  selected  to  decrease  the  Type  I error,  and  large  saziple 
sizes  made  the  probability  of  making  a Type  II  error  almost 
zero.  .All  tables  illustrating  statistical  results  give  infor- 
mation on  both  a = 0.01  and  a = 0.05  levels  of  significance,  so 
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that  the  reader  can  decide  how  critical  the  level  of  a = 0.01 
was  to  the  results  of  the  analysis. 

DETERMINATION  OF  APPROPRI.-ITE  5T.\TISTICAL  TEST 

C.12  Two  statistical  tests  were  selected  to  test  the  differences 
between  two  population  means.  The  first  test  was  a one-way  anal- 
ysis of  variance  (.A.NOV.A)  with  two  treatment  groups  (which  assumes 
that  both  populations  have  equal  variances).  The  second  test  was 
a two-tailed  t-test  (which  assumes  that  the  variances  are  un- 
equal). To  determine  the  appropriate  test,  the  F-ratio  was  used 
to  test  the  hypothesis  that  the  two  populations  had  equal  vari- 
ances against  the  alternative  hypothesis,  i.e.,  the  two  popula- 
tions have  unequal  variances).  To  facilitate  discussion  of  the 
three  statistical  tests  used,  the  basic  definitions  are  furnished 
below ; 

• ■''i  ” sample  sice  of  sample  i 

• = mean  of  sample  i 

• ^i  ” standard  deviation  of  sample  i 

• ^i  “ variance  of  sample  i. 

F-Ratio  Test 


C.15  The  F-ratio  test  is  appropriate  for  the  following  hypo- 
theses : 

• Hypothesis:  Both  populations  have  equal  variances 

• .-Xlternat  ive  hypothesis:  The  population  variances 

are  unequal. 


80 


PkESEARCH  INCORPOPIATSD 


I 


The  test  requires 


F = 


larger  sample  variance 
smaller  sample  variance 


to  be  calculated.  The  hypothesis  is  only  rejected  if 


F > F ^ (.N,-  1,  NS-  1) 

- (1  - ^ 

where  1 and  N-,-  1 represent,  respectively,  the  degrees  of 

freedom  associated  with  the  numerator  and  denominator.  The  F- 
value  is  found  by  using  the  degrees  of  freedom  and  (1  - %)  value 
to  enter  a cumulative  F-distribution  table. 

C.14  E.xamp  le . Using  the  data  from  the  Advanced  First  Term 
Avionics  (AFTA)  Course  (^performance  tests),  the  results  are: 

T1  Student  ISP-Type  Student 
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The  ANOVA  with  two  treatment  groups  tests  the  following  hypo- 
theses: 

• Hypothesis:  Both  populations  have  equal 
means 

• Alternative  hypothesis:  The  population 
means  are  unequal. 

The  hypothesis  is  rejected  only  if 

F > - a)  ’ ^^'2^ 

where  df^  and  df^  are  the  degrees  of  freedom.  The  F-value  is 
found  by  using  the  degrees  of  freedom  and  the  (1  - a)  value  to 
enter  the  cumulative  F-distribution  table. 

C.16  E.xamp le . Using  the  data  from  the  Aviation  Electrician's 
Mate  (AE)  performance  on  the  Third  Class  Advancement  E.\amina- 
tion  for  August  1975,  then: 

= number  of  ISD-type  graduates 
= 15 

= number  of  TI  graduates 
= 19 

Sum^  = sum  of  the  standard  scores  for  ISD-type 
graduates 

= 832.0 

Sum^  = sum  of  the  standard  scores  for  TI  graduates 

= 1,000.0 
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Sum  Sq  = sum  of  the  square  of  each  standard 
score 

= 101,334.0 

Total  SS  = 101,334  - = 2,621.53 


Treat  SS  = 


o4 

1832)'  ^ (1,000) 
15  19 


= 67.58 
o4 


Error  SS  = 2,621.55  - 67.38  = 2,554.15 


df  ^ = 1 
df,  = 2 


Treat  MS 
Error  MS 


67.38 


= 67.33 


= 79.82 


F = ^ 


j- 

6-’.5S 


79.82 


= 0.84 


and 


^(0.99)  .'.50, 


Therefore  the  hypothesis  is  accepted,  and  it  can  be  concluded 
that  there  is  no  difference  in  the  performances  of  the  ISD- 
type  and  TI  graduates  on  the  .August  1975  examination  at  the 
a = 0.01  leve 1 . 


Two-Tailed  t-Test 


C.17  The  two-tailed  t-test  is  used  to  test  the  hypothesis 
when  equal  variances  cannot  be  assumed. 

• Hypothesis:  Both  populations  have  equal 

means 

• .Alternative  hypothesis;  The  population  means 
are  unequal. 
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and  reject  the  hypothesis  if  the  absolute  value  of 

I 1 1 > t ’ . 

The  values  of  tj^  and  t,  are  found  by  entering  the  cumulative 
t-distribution  tables  with  (1  - y)  and  (N^  - 1)  or  (X,  - 1) 
degrees  of  freedom. 

C.18  Example.  .Again  using  the  data  from  the  AFTA  course  for 
the  performance  tests,  the  following  is  computed, 
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or 


|tl  = 4.047 


and  compute  t ' 


where 

and 


^(0.995)  ^ ' 


^2  ^(0.995) 


then 


(40.70  X 2.592  , 26.83  x 2.588 

\ 204 555 

/40.70  26.83\^; 

y 204  ■"  355  ] 


or 


t’  = 2.5909. 


Since  4.047  is  greater  than  2.5909,  the  hypothesis  is  rejected 
at  the  a = 0.01  level  and  the  alternative  hypothesis  is  accepted. 
Then  the  statement  can  be  made  that  the  ISD-type  student  had 
significantly  higher  scores  than  the  TI  student. 

SUMMARY 


C.19  To  make  the  statistical  decisions  required  for  this  study, 
the  following  procedure  was  used; 

• The  statistical  hypothesis  was  defined 

• A significance  level  of  = 0.01  was  selected 

• The  appropriate  statistical  test  was  determined 
using  the  F-ratio  test 

Equal  variances  use  A.N'OVA 

Unequal  variances  use  two-tailed  t-test 
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• The  hypothesis  was  tested 

• The  results  were  stated. 

C.20  There  are  many  other  statistics  that  could  have  been  used 
for  this  study,  but  the  results  would  have  been  similar.  Sample 
sizes  were  sufficiently  large  to  reduce  both  Type  1 and  Type  II 
errors  to  an  acceptable  level. 


